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PREFACE 

In  every  major  system  acquisition  the  project 
manager  must  review  and  assess  the  uncertainty 
confronting  the  acquisition.  How  this  uncer¬ 
tainty  is  perceived,  assessed,  and  approached 
has  not  been  determined.  Risk  and  uncertainty 
are  integral  aspects  of  the  manager's  decision 
making  role,  whether  the  decision  is  micro  or 
macro  and  it  requires  an  assessment  of  the 
scope  and  magnitude  of  uncertainty.  Yet, 
research  disclosed  that  decision  makers  in 
the  Department  of  Defense  and  the  private 
sector  seldom  give  the  concept  much  attention. 

There  is  a  need  to  develop  an  awareness  and  a 
better  unders tanding  of  the  state-of-the-art 
in  managing  risk  and  uncertainty.  The  need 
exists  particularly  in  DoD  for  proper  attention 
to  the  policy  established  for  the  identifi¬ 
cation  of  risk  in  DoD  Directive  5000.  1  and  DoD 
Instruction  5000.2.  These  require  management 
to  provide  estimates  that  identify  bands  of 
uncertainty  and  the  risks  for  technological 
and  production  milestones. 

It  was  an  awareness  of  this  lack  of  attention 
and  focus  that  led  to  the  first  Risk  and  Uncer¬ 
tainty  Conference  at  the  University  of 
Southern  California  in  February  1979.  This 
conference  was  adjudged  an  outstanding  suc¬ 
cess.  One  of  its  outputs  was  an  intensified 
awareness  of  the  fragmented  and  ad  hoc  ap¬ 
proach  that  existed  to  understanding  the 
subject  of  risk  and  uncertainty.  Thus,  the 
seeds  for  the  genesis  of  the  second  Risk  and 
Uncertainty  Conference  were  planted  in  Los 
Angeles.  The  staff  of  the  Air  Force  Business 
Research  Management  Center  and  faculty  at  the 
University  of  Southern  California  sensed  the 
urgency  to  continue  the  further  defining  and 
bounding  the  discipline  of  risk  and  uncer¬ 
tainty  and  thus  co-sponsor  this  second  con¬ 
ference  . 

The  conference  had  as  a  theme  or  overall  ob- 
iective:  "Define  the  Current  State-of- the- 

Art  in  Managing  Uncertainty  in  Major  Acqui¬ 


sition  Activities."  A  basic  approach  used 
for  the  conference  was  the  selection  of 
outstanding  professionals  with  the  expertise 
to  generate  and  present  significant  papers 
and/who  would,  by  their  personal  contributions 
as  panel  members,  be  able  fo  place  initial 
boundaries  around  this  emerging  field  of 
knowledge.  The  idea  of  having  experts, 
theoretical  and  practical,  from  industry, 
government  and  academe  helped  pace  the 
organization  and  execution  of  the  conference. 

The  conference  theme  and  subject  of  risk  and 
uncertainty  was  divided  into  four  areas  (or 
sections):  applications,  taxonomic  concepts, 

methodologies,  and  assessments.  The  appli¬ 
cations  area  was  covered  first  to  identify  and 
intensify  the  problems  associated  with  the 
application  of  risk  and  uncertainty  concepts. 
The  four  areas  then  served  as  the  basis  for 
conducting  the  panels.  The  panels  in  each  of 
the  areas  had  additional  objectives,  including 
the  preparation  of  recommendations.  The  pan¬ 
els  met  the  workshop  objectives,  and  made  a 
significant  contribution  to  the  conference. 

The  panel  format  was  based  on  using  a  small 
group  approach.  Outstanding  theorists,  and 
practitioners  were  invited  to  present  papers, 
to  chair  panels  and  to  serve  in  appropriate 
other  roles.  Each  paper  was  presented  to  a 
panel  composed  of  ten  to  fifteen  individuals. 
Thus,  it  took  the  form  of  a  seminar.  The 
panel  chairmen  served  as  moderators  and  later 
summarized  the  panel  discussion  at  a  plenary 
session. 

This  interaction,  in  small  groups,  on  a  well 
defined  subject  for  a  lengthy  period  of  time 
(over  two  hours)  led  to  a  much  greater  depth 
of  discussion,  personal  involvement,  exchange 
of  ideas  and  cross  fertilization  than  occurs 
in  a  typLcal  professional  meeting.  The 
response  was  well  beyond  our  exportations. 

The  people  who  attended  came  with  enthusiasm 
and  dedication  to  the  task,  they  interacted 
and  they  hammered  out  the  general  beginnings 
of  a  new  discipline  which  has  at  its  focus 
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risk  and  uncertainty. 

The  initial  thrust  for  the  conference  was 
established  in  a  penetrating  manner  by  the 
keynote  speaker.  Brig.  Gen.  William  Thurman, 
Commandant  of  the  Defense  Systems  Management 
Co-lege.  He  reviewed  the  sophisticated 
weapon  systems  that  will  be  acquired  in  the 
1980’s,  and  the  levels  of  uncertainty  that 
will  surely  be  associated  with  them  in  terms 
of  cost,  schedule  and  performance.  He 
placed  the  problem  of  improving  management 
of  risk  and  uncertainty  squarely  on  the 
shoulders  of  the  conference  participants. 

While  the  published  papers  reflect  the  ex¬ 
perience  and  research  effort  of  the  authors, 
it  is  difficult  to  adequately  recognize  the 
dynamic  interaction  and  intellectual  stimu¬ 
lation  that  occurred  during  the  panel  dis¬ 
cussions  and  after  the  presentation  of 
each  paper.  Those  serving  as  panel  chair¬ 
men  in  each  area  (or  section)  met,  selected 
a  recording  secretary  and  reported  on  these 
discussions,  suggestions,  and  the  recom¬ 
mendations  of  each  panel.  The  recorders 
then  presented  this  information  to  the 
final  general  session.  Again,  there  was 
lively  discussion. 

The  written  reports  and  notes  of  the  panel 
chairmen  have  been  synthesized  and  combined 
into  a  summary  for  each  area  or  section  of 
this  proceeding.  The  range  and  thrust  of 
the  recommendations  from  these  panels  are 
an  indication  of  the  momentum  that  was 
attained.  The  seriousness  and  dedication 
of  participants  to  continue  their  study 
and  research  is  also  evident  from  the 
recommendations  for  future  activities. 

One  recommendation  receiving  almost  unan¬ 
imous  recognition  was  the  need  for  defini¬ 
tion.  The  identification  of  semantic 
differences  from  the  mathematical  to  the 
practical  usage  of  terms  in  disucssions 
was  most  apparent  and  frequently  frustra¬ 
ting.  The  same  terms  were  used  for  many 
very  different  concepts  and  methodologies. 

This  problem  leads  to  the  next  major  recom¬ 
mendation  for  the  need  of  further  defini¬ 
tional  categories  and  developing  an  ap¬ 
propriate  texonomy  as  essential  to  the 
order  and  progress  of  the  discipline. 

Without  well  defined  and  understood  termi¬ 
nology  the  semantics  required  to  translate 
or  integrate  applications  into  state-of- 
the-art  via  a  taxonomy  and  flush  out  the 
skeletal  framework  will  be  extremely 
dill  I  cult. 

There  were  a  number  of  specific  recommen¬ 
dations  made  by  the  panels.  They  constitute 
a  substantive  list  of  problems  and  issues 
for  further  study  and  research.  These  are 


shown  at  the  end  of  each  section  of  papers  in 
the  proceedings. 

Above  all  we  believe  that  the  conference  es¬ 
tablished  a  technical  and  management  base¬ 
line  upon  which  to  build.  Thus,  in  closing 
our  challenge  to  the  participants  who  will  now 
study  the  conference  results  and  to  those 
who  study  these  proceedigns  is  that  they 
accept  the  challenge  of  the  research,  and 
experimental  applications  needed  to  meet  an 
ever  increasingly  complex  and  turbulent  ac¬ 
quisition  environment.  The  resources  must 
be  applied  to  this  task  and  contracting 
agencies  must  focus  on  how  best  to  integrate 
these  concepts  into  more  meaningful  con¬ 
tractual  expectations. 


As  a  final  note,  two  dedicated  individuals,  Ed 
Cochran  and  Capt.  William  Glover,  who  to  a 
large  extent  were  responsible  for  the  concept 
of  this  second  conference  are  no  longer  with 
us.  It  is  our  fond  hope  that  what  we  accom¬ 
plished  will  serve  as  a  fitting  memorial  to 
two  incredible  people. 

May  1981 

Col.  Martin  D.  Martin 
Dr.  Alan  J.  Rowe 
Dr.  Herold  A.  Sherman 
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BRIGADIER  GENERAL  WILLIAM  E.  THRUMAN 

origadier  General  William  E.  Thurman  is  com- 
madant  of  the  Defense  Systems  Management  Col¬ 
lege,  Fort  Belvoir,  Virginia. 

He  graduated  from  the  U.S.  Naval  Academy,  An¬ 
napolis,  Maryland,  in  1954  with  a  bachelor  of 
science  degree  and  was  commissioned  as  a  sec¬ 
ond  lieutenant  in  the  U.S.  Air  Force.  He 
earned  a  master  of  science  degree  in  aeronau¬ 
tical  engineering  from  the  Air  Force  Institute 
of  Technology,  Wright-Patterson  Air  Force  Base, 
Ohio,  in  1962  and  a  master  of  administration 
degree  in  management  engineering  from  the 
George  Washington  University,  Washington,  D.C., 
in  1971.  He  completed  Squadron  Officer  School 
at  Maxwell  Air  Force  Base,  Alabama,  in  1959; 
the  Army  Command  and  General  Staff  College  at 
Fort  Leavenworth,  Kanses ,  in  1965;  the  Indus¬ 
trial  College  of  the  Armed  Forces  at  Fort  Les¬ 
ley  J.  McNair,  Washington,  D.C.,  in  1975;  and 
the  Stanford  University  executive  program  in 
1977.  He  also  attended  the  University  of  Ken¬ 
tucky  and  Ohio  State  University. 

Assigmnets  in  research  and  development  have 
included  tours  of  two  years  with  the  Aero¬ 
space  Research  Laboratories  at  Wright-Pat¬ 
terson  Air  Force  Base,  one  year  with  Air  De¬ 
fense  Command  at  Ent  Air  Force  Base,  Colo¬ 
rado,  two  years  in  Headquarters  U.S.  Air  Force 
at  the  Pentagon,  two  years  with  the  National 
Aeronautics  and  Space  Council  staff  in  the 
Executive  Offices  of  the  president,  and  6*5 
years  with  the  Aeronautical  Systems  Division 
at  Wright-Patterson  Air  Force  Base. 
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Assigned  to  the  Aeronautical  Systems  Division 
in  August  1971,  General  Thurman  became  a 
charter  member  of  the  Prototype  Program 
Office  as  its  assistant  director.  He  was 
named  the  deputy  for  prototypes  in  June  1973, 
deputy  for  air  combat  fighter  in  May  1974, 
deputy  for  F-16  in  January  1975  and  deputy 
for  engineering  in  May  1976.  He  received 
the  Air  Force  Associations  Meritorious  Award 
for  Program  Management  in  1976, 

In  May  1978  General  Thurman  was  assigned  to 
the  Electronic  Systems  Division  at  Hanscom 
Air  Force  Base,  Massachusetts,  as  deputy 
for  control  and  communications  systems. 

General  Thurman  began  his  current  duty  in 
July  1979. 

MR.  RICHARD  D.  ABEYTA 

Mr.  Abeyta  has  spent  the  last  four  years 
researching,  teaching,  and  applying  risk 
and  uncertainty  analysis  in  the  Army  material 
acquisition  process.  He  has  researched  and 
published  several  studies  applying  Operations 
Research  and  Decision  Analysis  to  the  Army 
logistics  process  and  has  coauthored  a  hand¬ 
book  on  Decision  Risk  Analysis.  He  is 
currently  on  the  graduate  night  school  staff 
of  the  Florida  Institute  of  Technology  and 
Chapman  College,  teaching  Operations 
Research.  He  is  a  member  of  Alpha  Pi  Mu 
and  Tau  Beta  Pi  honorary  societies.  Mr. 

Abeyta  is  Chief  of  the  Decision  Risk 
Analysis  Committee  in  the  Systems  and 
Cost  Analysis  Department  of  the  U.S.  Army 
Logistics  Management  Center  (ALMC). 

DR.  DAVID  N.  BURT 

David  N.  Burt  currently  is  lecturer  in 
marketing  and  logistics  at  San  Jose  State 
University.  Dr.  Burt  is  a  frequent  guest 
lecturer  and  speaker  in  this  country  and 
abroad  on  topics  in  the  areas  of  acquisition 
management,  logistics,  production,  and  pur¬ 
chasing.  Dr.  Burt  received  his  Ph.D.  from 
Stanford  University  in  1971.  Since  that  time, 
he  has  taught  at  the  two  graduate  schools  of 
the  Department  of  Defense  and  been  a  con¬ 
sultant  to  several  DOD  and  private  organi¬ 
zations.  He  has  authored  some  20  articles 
in  the  areas  of  acquisition,  logistics,  and 
long  range  planning.  His  current  efforts 
are  directed  to  a  book  on  the  material 
acquisition  process  in  the  private  sector. 

MAJ  RICHARD  F.  DeMONG  (USAF  Reserve) 

Major  DeMong  is  an  Associate  Professor  of 
Commerce  at  the  Mclntire  School  of  Commerce, 
University  of  Virginia.  He  is  a  Doctor  of 
Business  Administration  in  Finance  and 
Management  Science  (University  of  Colorado). 
He  has  published  articles  in  the  American 
Journal  of  Small  Business,  Journal  of 


General  Management,  National  Contract 
Management  Quarterly  Journal,  and  the 
Defense  Management  Journal.  Also,  he  is  a 
member  of  several  national  financial 
associations.  Major  DeMong  is  a  Research 
Associate  (USAF  Reserves)  at  the  Air  Force 
Business  Research  Management  Center,  Wright- 
Patterson  AFB,  Ohio. 

DR.  WARD  EDWARDS 

After  spending  15  years  at  the  University 
of  Michigan,  Dr.  Ward  Edwards-  moved  to  the 
University  of  Southern  California  in  1973. 
He  is  now  Director  of  the  Social  Science 
Research  Institute  and  Professor  of 
Psychology  and  of  Industrial  and  Systems 
Engineering  at  USC.  The  Institute  that 
he  directs  conducts  large  scale  research 
programs  applying  scientific  tools  and 
methods  to  problems  such  as  evaluation  and 
improvement  of  criminal  justice  systems, 
techniques  for  resolving  criminal  and 
civil  disputes  without  recourse  to  the 
courts,  use  of  geographically  coded 
records  for  planning  and  social  program 
evaluation,  differential  equation  models 
of  social  phenomena,  biological  and  social 
determiners  of  schizophrenia  and  crimi¬ 
nality,  physicaUand  social  tradeoffs 
among  alternative  energy  sources,  and  the 
like. 

Dr.  Edwards  is  a  psychologist  by  training, 
and  his  personal  research  interests  are 
mostly  concerned  with  the  methods  and 
a p p 1 i cations  of  decision  analysis  with 
behavioral  decision  theory.  He  helped 
introduce  Bayesian  ideas  into  psychological 
research,  primarily  from  the  point  of 
view  that  uses  Bayes's  Theorem  as  a  model 
of  human  inference  processes.  His  current 
research  is  concerned  with  the  character¬ 
istics  of  multiattribute  utility  models, 
with  elicitation  methods  appropriate  to 
their  use,  and  with  their  application  to 
social  program  and  evaluations.  His 
research  has  always  attempted  to  blend 
abstractions  with  applications;  most  of  it 
is  set  in  or  relevant  to  solution  of  some 
real-world  decision  problem. 

MR.  JOHN  S.  W.  FARGHER,  JR. 

After  his  graduation  from  the  Montana 
College  of  Mineral  Science  and  Technology 
in  1969,  Mr.  Fargher  began  his  Department 
of  Army  civilian  career  as  a  Production 
Design  Engineer  at  the  Red  River  Army 
Depot  Intern  Training  Center.  He  was 
graduated  from  the  center  in  1971  with  a 
certificate  as  a  Production  Design 
Engineer,  and  also  received  a  master  of 
engineering  degree  in  industrial  engi¬ 
neering  from  Texas  A  &  M. 
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His  initial  assignment  was  the  Systems 
Engineer  for  the  Small  Caliber  Ammunition 
Moderization  Program  (SCAMP),  Frankford 
Arsenal,  where  he  directed  the  development 
of  the  Ballistic  Test  System,  which  is  now 
the  Army  standard  for  ammunition  proof  and 
acceptance.  In  1973,  he  was  Chief  of  the 
Production  Planning  and  Control  Division, 

U.S.  Naval  Ammunition  Depot,  Hawthorne,  Nev., 
directing  the  workloading  of  the  Ordinance 
Department.  In  1974,  he  became  Lead 
Mechanical  Engineer  with  Rodman  Laboratory, 
Rock  Island,  Ill.,  and  attended  PMC  74-2 
at  the  Defense  Systems  Management  College. 

In  1975,  Mr.  Fargher  was  Industrial  Engi¬ 
neer  with  the  Iranian  Aircraft  Project 
Manager's  Field  Office,  Teheran,  Iran, 
managing  the  Bell  Helicopter  International 
contracts  for  establishment  of  a  helicopter 
logistics  system  for  the  Government  of  Iran. 
After  completion  of  his  tour  in  1977,  Mr. 
Fargher  was  Deputy  Chief  of  the  Materiel 
Systems  Development  Division,  Directorate 
of  Combat  Developments,  U.S.  Army  Trans¬ 
portation  School,  Fort  Eustis,  Va.  He 
joined  the  staff  of  the  Defense  Systems 
Management  College  in  December  1978.  Mr. 
Fargher  also  completed  a  second  masters 
degree,  an  M.S.  in  systems  management 
(logistics  option)  in  1978,  started  while 
a  student  at  DSMC  in  1974. 

Since  joining  the  DSMC  faculty,  Mr. 

Fargher  has  managed  the  research  for  A 
Guide  for  the  Management  of  Joint  Service 

Programs  for  the  Joint  Logistics  Commanders, 
and  a  NATO  RSI  Acquisition  Guide  for  the 
Office  of  the  Secretary  of  Defense,  as  well 
as  developing  courses  and  teaching  in  PMC, 
SAFMC,  and  MPMC.  He  is  also  the  Executive 
Secretary  of  the  Defense  Acquisition 
Research  Element  (DARE)  for  coordinating 
acquisition  research  with  DOD  and  FAI. 

His  current  position  is  Professor  of 
Acquisition/Program  Management,  Research 
Division,  Department  of  Research  and 
Information,  DSMC. 

MR.  ALFRED  M.  FEILER 

As  President  of  Log/An,  Inc.  since  its 
founding  in  1965,  Mr.  Feiler  has  directed 
the  firm's  operations,  including  develop¬ 
ment  of  its  proprietary  computer-based 
project  risk  management  tools  which  have 
received  widespread  acceptance  in  industry 
and  government.  Such  techniques  are  now 
in  worldwide  use  of  projects  totalling 
over  $35  billion  in  value. 

Mr.  Feiler' s  experiences  have  centered  on 
the  systems  approach  to  solution  of 
complex  management  problems  in  the  fields 
of  research  and  development,  engineering, 
and  construction. 


Mr.  Feiler  recently  retired  from  the 
academic  staff  of  the  University  of 
California  (Los  Angeles),  where,  for  16 
years,  he  directed  research  and  develop¬ 
ment  in  the  management  sciences  for  several 
government  agencies. 

Prior  to  that,  Mr.  Feiler  was  Area  Vice 
President  of  Pneumodynamics  (now  Pneumo) 
Corporation,  and  earlier  was  Division 
Manager  of  Pneumodynamics '  Advanced  Systems 
Development  Division,  El  Segundo,  Cali¬ 
fornia.  During  this  time,  he  successfully 
applied  the  systems  approach  to  development 
of  a  line  of  products  for  industrial  and 
military  uses.  Previously,  he  was  Chief 
of  Advanced  Systems  Research,  Aircraft 
Division,  Hughes  Tool  Company,  where  he 
was  primarily  concerned  with  systems 
analysis  of  logistics  and  transportation 
pro bl  ems . 

Mr.  Feiler  has  authored  numerous  reports 
and  papers  on  simulation,  project  risk 
management,  and  systems  analysis;  and 
has  served  on  several  government  committees 
and  advisory  boards  in  connection  with  his 
specialty  fields  of  interest.  He  is  a 
registered  Professional  Engineer  in 
California  and  New  York.  Mr.  Feiler  re¬ 
ceived  his  B.S.  and  M.S.  degrees  in  Civil 
Engineering  from  Carnegie  Tech  in  1941  and 
1942,  respecti vel y . 


DR.  PETER  C.  GARDINER 

Dr.  Peter  C.  Gardiner,  newly  appointed  as 
Associate  Chairman  of  the  General  Systems 
faculty,  is  an  Associate  Professor  at  the 
University  of  Southern  California's  In¬ 
stitute  of  Safety  and  Systems  Management. 

Prior  to  joining  the  USC  faculty,  Dr. 
Gardiner  served  as  Systems  Control  Of¬ 
ficer  and  Operations  Project  Officer  for 
the  Defense  Communication  Agency,  Euro¬ 
pean  Area,  while  serving  in  the  Navy 
from  1963-1969. 

He  is  the  author  of  numerous  professional 
publications  in  such  subject  areas  as 
systems  dynamics  modeling  and  simulation, 
multi-attribute  utility  measurement,  and 
managing  complex  systems. 

Dr.  Gardiner,  who  was  formerly  a  teaching 
Fellow  in  the  Industrial  Engineering  De¬ 
partment  at  the  University  of  Michigan, 
belongs  to  the  Society  for  Computer 
Simulation,  The  World  Futures  Society, 
the  Society  of  Logistics  Engineers  and 
the  Institute  of  Management  Sciences. 

Since  coming  to  the  University,  he  has 
also  been  continuously  active  in  analy¬ 
tical  research  problems,  principally  in 
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his  role  as  a  research  associate  at  USC's 
Social  Science  Research  Institute. 

Dr.  Gardiner  was  awarded  his  BS  in  Elec¬ 
trical  Engineering,  MS  in  Systems  Man¬ 
agement  and  his  Ph.D.  in  Urban  Studies, 
all  from  the  University  of  Southern 
Cal  i  fornia . 


DR.  GEORGE  P.  JONES 

George  P.  Jones  joined  the  University  of 
Southern  California  in  1970  as  chairman  of 
the  Systems  Management  Department  at  the 
Institute  of  Safety  &  Systems  Management. 

At  the  Institute,  Jones  participated  in 
promoting  and  opening  new  study  centers. 

He  was  Principal  Investigator  on  funded 
research  studies  such  as  a  "Risk  Analysis 
of  Hazardous  Materials  Transportation" 
study  for  DOT'S  Office  of  Hazardous 
Materials  (OHM).  He  also  led  a  research 
study  for  the  development  of  a  safety  plan 
for  the  Urban  Mass  Transporta tion  Admini¬ 
stration  (UMTA).  Jones  has  functioned  as 
Director  of  the  Institute’s  Research 
Center  and  Acting  Executive  Director  of 
ISSM's  worldwide  graduate  level  program. 

In  addition  to  his  administrative  duties 
Jones  holds  a  joint  appointment  as 
Associate  Professor  in  Systems  Management 
and  Industrial  and  Systems  Engineering 
(ISE).  He  has  consulted  for  organi zations 
such  as  R.A.N.D.,  application  of  management 
science  techniques,  statistics  and  risk 
analysi  s . 

Formerly,  Jones  had  a  career  ranging  from 
research  and  development  engineering 

through  management  of  systems  analysis 
projects  at  North  American  Rockwell  and  other 
industrial  organi zations .  He  served  as 
Chief  Engineer  and  Director  of  Advanced 
Systems  during  the  inception  of  the  FJ 
series,  RA-5,  X-15,  0V-10,  B-l  Aircraft 
and  other  missile  and  space  systems. 

Jones  earned  his  Ph.D.  degree  in  Industrial 
and  Systems  Engineering,  with  concentration 
in  the  management  sciences,  from  Ohio  State 
University;  a  M.Sc.  degree  from  the  same 
university  in  Mechanical  Engineering, 
with  specialization  in  thermodynamic s  and 
heat  transfer,  and  a  B.M.E.  degree  from 
The  Cooper  Union  College  of  Engineering  in 
New  York  City,  his  native  town. 


DR.  CRAIG  W.  KIRKWOOD 

Craig  W.  Kirkwood  is  a  member  of  the  de¬ 
cision  analysis  group  at  Woodward-Cl  yde 
Consultants,  San  Francisco,  California 


94111.  He  has  extensive  experience  direc¬ 
ting  and  participating  in  studies  of  risk 
and  uncertainties  associated  with  the 
acquisition  of  major  systems.  This 
experience  includes  site  selection  for 
electric  power  generation  facilities  and 
nuclear  waste  reposi tori es ,  analysis  of 
public  risks  associated  with  a  proposed 
liquefied  natural  gas  terminal,  evalu¬ 
ation  of  proposals  for  solar  photovol¬ 
taic  demonstration  projects,  and  a  variety 
of  other  studies.  He  has  utilized  de¬ 
cision  analysis  on  these  and  other  prob¬ 
lems,  and  has  published  articles  on  both 
theoretical  and  applied  aspects  of  de¬ 
cision  analysis.  He  received  S.B.,  S.M., 
E.E.  and  PH.D.  degrees  from  the  Mas¬ 
sachusetts  Institute  of  Technology. 

MR.  GERALD  A.  KLOPP 

Served  as  a  Production  Engineer  at  the 
Electronics  Command,  Ft.  Monmouth,  NJ; 
an  Operations  Research  Analyst  at  the 
Comptroll  er's  Office,  Ft.  Ritchie, 

MD;  Operations  Research  Analyst,  Sys¬ 
tems  and  Cost  Analyst  Department,  US 
Army  Logistics  Management  Center,  FT 
Lee,  Va;  and  Operations  Analyst  in  the 
Technical  Design  and  Validation  Branch 
of  the  Operations  Analysis  Directorate 
of  the  US  Army  Logistics  Center,  FT 
Lee,  VA.  Currently  serving  as  the 
Statistician  at  the  Army  Procurement 
Research  Office,  Ft  Lee,  VA.  Experi¬ 
ence  in  Operations  Research  include 
a  Hybrid  Computer  Simulation  of  a  large 
scale  production  system;  senior  in¬ 
structor  and  Course  Director  of  the 
ORSA  Military  Applications  Course  I; 
publication  of  7  articles  on  computer 
modeling  and  simulation;  and  several 
interactive  computer  programs  for 
statistical,  economic,  and  Operations 
Research  applications.  Education 
includes  a  BS  in  Electrical  Engineering 
from  Michigan  State  University  (1972) 
and  ME  in  Industrial  Engineering  from 
Texas  A&M  University.  Production 
System  Analysis,  and  Systems  Analysis 
for  the  Florida  Institute  of  Tech¬ 
nology  at  Ft  Lee,  VA. 

MR.  GEORGE  T.  KRAEMER 

Manager  of  Engineering  Control  and  Risk 
Analysis  at  Boeing  Vertol  Company 
under  the  Director  of  Engineering 
Operations.  Graduated  from  the  Uni¬ 
versity  of  Pennsylvania  with  a  degree 
in  mathematics  (B.S.  1951)  and  in  1970, 
received  a  Master  of  Business  Admini¬ 
stration  (statistics)  from  Temple 
University.  Member  of  the  American 
Helicopter  Society.  Thirty  years  ex- 
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perience  in  engineering  business  management, 
subcontract  management,  and  master  planning  in 
aircraft  and  electronic  industries.  Also  have 
written  several  papers. 

DR.  DAVID  V.  LAMM 

Dr.  David  V.  Lamm  is  currently  an  Assistant 
Professor  of  Administrative  Sciences  at  the 
Naval  Postgraduate  School  in  Monterey,*  Calif¬ 
ornia  where  he  instructs  military  officers 
and  civilians  in  the  fields  of  Acquisition, 
project  management,  and  contracting. 

He  holds  a  BA  in  History  and  Political  Sci¬ 
ence  from  the  University  of  Minnesota,  a  MBA 
in  Procurement  and  Contracting  from  the  George 
Washington  University,  and  a  DBA  in  Procure¬ 
ment  and  Production,  and  Science  and  Techno¬ 
logy  Management  from  The  George  Washington 
University. 

He  is  currently  serving  on  active  duty  as  a 
Navy  Commander.  He  has  held  various  tours 
both  afloat  and  ashore  in  the  Navy.  Ac¬ 
quisition  and  contracting  tours  include  duty 
at  Headquarters,  Defense  Contract  Adminis¬ 
tration  Services  (DCAS)  and  Headquarters, 

Naval  Air  Systems  Command  where  he  served  as 
Assistant  Director,  Missile  Weapon  Systems 
Purchase  Division. 

He  is  a  member  of  the  National  Contract  Man¬ 
agement  Association  (NCMA)  and  is  a  Certi¬ 
fied  Professional  Contracts  Manager  (CPCM) . 

He  has  participated  in  the  CPCM  program 
since  1975  as  the  first  Chairman  of  the  CPCM 
Examination  Board  and  currently  as  an  exami¬ 
nation  grader.  He  has  been  a  frequent 
speaker  and  panelist  at  various  meetings 
and  workshops.  He  was  the  First  Place  Win¬ 
ner  of  the  1978  Blanche  Witte  Award  and  is 
currently  director  of  the  Monterey  Peninsula 
NCMA  Chapter. 

DR.  RICHARD  LORETTE 

Dr.  Richard  Lorette  is  a  member  and  Fellow  of 
NCMA  and  editor  of  the  National  Contract 
Management  Journal;  he  has  held  management 
positions  in  the  Air  Force  Systems  Command 
(Weapons  Acquisition;  B-52  SPO  and  C/KC-135 
SPO)  and  Air  Force  Institute  of  Technology 
prior  to  his  retirement  from  the  Air  Force  in 
1973.  Dr.  Lorette  has  taught  courses  in  Pro¬ 
curement,  Systems  Management,  and  Program  Man- 
agement  and  published  several  articles  in  the 
general  area  of  Procurement  and  Systems  Ac¬ 
quisition.  Recently,  he  completed  a  one-year 
consulting  assignment  (from  the  University  of 
Southern  California)  to  the  Federal  Acquisi¬ 
tion  Institute,  studying  major  acquisition 
issues  and  problems  in  the  federal  civil  ag¬ 
encies.  He  is  a  1950  Graduate  of  West  Point 
has  an  MBA  in  Engineering  Administration 
(AFIT) ,  a  Doctorate  in  Business  Administra¬ 
tion  from  the  Harvard  University  Graduate 
School  of  Business  Administration  and  is 


currently  Professor  of  Management  Science  in 
the  College  of  Business  at  Ball  State  Univer¬ 
sity  in  Muncie,  Indiana. 

COLONEL  MARTIN  D.  MARTIN 

Colonel  Martin  currently  serves  as  the  Execu¬ 
tive  Director  of  the  Air  Force  Business  Re¬ 
search  Management  Center,  Headquarters  USAF, 
Wright-Patterson  AFB ,  Ohio. 

Colonel  Martin  holds  a  Bachelor  of  Science  De¬ 
gree  in  Industrial  Education  from  the  Louisi¬ 
ana  State  University  and  both  a  Master  of  Busi¬ 
ness  Administration  Degree  and  a  Doctor  of 
Philosophy  Degree  in  Financial  Management, 
Marketing,  Systems  Management,  and  Economics 
from  the  University  of  Oklahoma. 

He  has  extensive  experience  in  the  acquisition 
and  contracting  areas  as  a  manager.  In  the 
contract  administration  area  he  has  served  as 
both  the  Chief  of  Management  Engineering  and 
the  Assistant  Chief,  Office  of  Planning  and 
Management,  of  a  major  Defense  Contract  Admin¬ 
istration  Services  Region  and  as  the  Chief  of 
a  Defense  Contract  Administration  Services  Of¬ 
fice.  At  major  command  level,  USAF,  duties 
included  the  function  of  a  procurement  manage¬ 
ment  staff  officer  in  the  area  of  weapon  sys¬ 
tems  acquisition  and  procurement  pricing  po¬ 
licy.  He  also  served  as  a  Deputy  Program  Man¬ 
ager.  He  has  served  as  a  Director  of  Manu¬ 
facturing  for  a  major  Systems  Program  Office 
and  the  Director  of  Contracting  for  Wright- 
Patterson  AFB.  His  Air  Force  functional  ex¬ 
perience  has  included  duty  as  a  procuring  con¬ 
tracting  officer,  administrative  contracting 
officer,  and  a  price  and  cost  analyst  in  the 
contracting  and  acquisition  operational  area. 

In  the  education,  research,  and  consulting 
areas,  Colonel  Martin  served  for  four  years  as 
Associate  Professor  of  Logistics  Management 
and  Department  Chief,  Functional  Management 
Department,  Graduate  Education  Division,  School 
of  Systems  and  Logistics,  Air  Force  Institute 
of  Technology.  He  is  at  present  an  Adjunct 
Associate  Professor  of  Marketing,  Wright  State 
University,  Dayton,  Ohio.  He  has  also  taught 
and  delivered  lectures  at  Stanford  University, 
University  of  Southern  California,  University 
of  Oklahoma,  Wittenberg,  Sinclair  Community 
College,  and  the  University  of  Dayton.  He  has 
consulted  and  conducted  seminars  for  indus¬ 
trial,  government,  and  military  management  in 
the  United  States  and  the  United  Kingdom. 

Colonel  Martin  is  a  member  of  Phi  Kappa  Phi, 
national  education  honor  society;  Omicron  Del¬ 
ta  Kappa,  national  scholarship  honor  society; 
Beta  Gamma  Sigma,  national  business  adminis¬ 
tration  honor  society;  and  others.  His  pro¬ 
fessional  and  business  organizations  include 
the  Project  Management  Institute,  the  Ameri¬ 
can  Finance  Association,  the  National  Contract 
Management  Association,  and  the  Society  of 
Logistics  Engineers. 
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LT  COL  JACK  L.  McCHESNEY 

Lt  Col  Jack  L.  McChesney  is  an  Assistant  Prof¬ 
essor  of  Logistics  Management  in  the  Contract¬ 
ing  Management  Department,  School  of  Systems 
and  Logistics,  Air  Force  Institute  of  Techno¬ 
logy.  He  focuses  his  teaching  in  the  specia¬ 
lized  areas  of  the  Graduate  Contracting  Major. 

Lt  Col  McChesney  holds  a  Bachelor  of  Business 
Administration  Degree  from  North  Texas  State 
University,  A  Master  of  Business  Administra¬ 
tion  Degree  from  the  University  of  Texas  at 
Austin,  and  a  Doctor  of  Business  Administra¬ 
tion  Degree  from  George  Washington  University. 
His  Air  Force  experience  includes  assignments 
as  Base  Procurement  Officer  at  England  AFB , 
Louisiana,  and  Cannon  AFB,  New  Mexico.  In 
that  capacity,  he  made  purchases  and  admin¬ 
istered  contracts  for  supplies,  services  and 
construction.  He  served  as  a  staff  contract 
administrator  in  the  HQ  Defense  Contract  Ad¬ 
ministration  Services  at  Cameron  Station,  Al¬ 
exandria,  Virginia.  While  in  DCAS ,  Lt  Col 
McChesney  was  responsible  for  developing,  im¬ 
plementing  and  monitoring  policies  and  proce¬ 
dures  for  contract  administration  on  a  DOD- 
wide  basis.  His  most  recent  assignment  was 
with  the  HQ  Air  Force  Systems  Command,  And¬ 
rews  AFB,  where  he  worked  in  the  Directorate 
of  Procurement  Data  Systems,  DCS /Procurement 
and  Manufacturing.  During  that  tour  of  duty, 
he  participated  in  the  development  and  im¬ 
plementation  of  a  command-wide  acquisition 
management  information  system.  Lt  Col  Mc¬ 
Chesney  is  a  member  of  Beta  Gamma  Sigma,  Phi 
Kappa  Phi,  Phi  Eta  Sigma  and  Blue  Key  nation¬ 
al  honorary  societies  and  a  Fellow  member  of 
the  National  Contract  Management  Association. 

DR.  GERALD  R.  McNICHOLS 

Dr.  McNichols  is  President  and  Technical  Dir¬ 
ector  of  Management  Consulting  &  Research,  Inc. 
(MCR)  a  firm  which  specializes  in  resource 
analysis  studies  including  Life  Cycle  Cost, 
Design-to-Cost ,  Uncertainty  Analysis  and  Risk 
Assessment,  manpower,  operations  research,  and 
economic  analyses.  He  is  co-author  of  a  text- 
bo°k>  Operations  Research  in  Decision  Making 
(1975),  past  editor  of  the  Cost-Effectiveness 
Newsletter,  and  past-president  of  the  Washing- 
ton  Operations  Research  Council.  For  the  past 
17  years  he  has  been  developing  and  applying 
resource  analysis  techniques  in  government 
and  industry.  He  was  a  Vice  President  of  J. 
Watson  Noah  Associates,  Inc.  (cost  analysis 
consultants)  and  a  Vice  President  and  Dir¬ 
ector,  Defense  and  Technology  Programs  Divi¬ 
sion  of  GENTECH,  Inc.  (management  and  inform¬ 
ation  system  consultants).  During  his  nine 
years  of  government  service  he  worked  for 
the  Deputy  Assistant  Secretary  of  Defense  for 
Resource  Analysis,  Deputy  Assistant  Secretary 
of  Defense  for  Strategic  Programs  (both  under 
the  Assistant  Secretary  for  Systems  Analysis), 
and  in  the  Office  of  the  Chief,  Operations 


Analysis,  HQ  USAF.  While  in  DoD,  Dr.  Mc¬ 
Nichols  directed  efforts  to  formulate  fiscal 
guidance  for  the  annual  service  budgets,  evalu¬ 
ated  forces  and  systems  requirements  as  pro¬ 
posed  by  the  Military  Services  using  cost- 
effectiveness  tools  and  models  as  an  advisor 
to  the  DSARC,  and  developed  parametric  cost 
relationships  and  Life  Cycle  Costs  as  a  mem¬ 
ber  of  the  Cost  Analysis  Improvement  Group 
(CAIG).  He  introduced  cost  uncertainty  anal¬ 
ysis  to  the  DSARC  Milestone  Reviews  in  197  2. 

He  is  also  Professor  of  Engineering  at  George 
Washington  University. 

He  holds  a  Doctor  of  Science  degree  in  Eng¬ 
ineering/Operations  Research  with  a  minor  in 
economics  and  has  authored  numberous  articles/ 
research  papers  in  the  fields  of  resource 
analysis  and  operations  research. 

DR.  HERBERT  MOSKOWITZ 

Herbert  Moskowitz,  Ph.D.  UCLA,  is  Professor 
of  Management  at  the  Krannert  Graduate  School 
of  Management,  Purdue  University.  His  publi¬ 
cations  include  papers  in  such  journals  as 
Management  Science,  Operations  Research,  Omega, 
Academy  of  Management,  IEEE  Transactions  on 
Engineering  Management,  AIIE  Transactions,  De¬ 
cision  Sciences,  and  Organizational  Behavior 
and  Human  Performance.  He  is  a  vice-president 
and  a  member  of  the  Executive  Board  of  AIDS 
and  a  member  of  TIMS,  and  ORSA. 

MR.  MICHAEL  D.  RICH 

Mr.  rich  is  currently  Director,  Resource  Man¬ 
agement  Program  at  the  Rand  Corporation  in 
Santa  Monica,  CA.  He  is  also  directing  a  study 
of  prospective  Air  Force  participation  in  Mul¬ 
tinational  coproduction  programs.  Topics  be¬ 
ing  studied  include  cost  and  schedule  implica¬ 
tions  of  various  types  of  coj Xnhorati vt»  arrange¬ 
ments,  the  prohiem  of  third-country  sales,  and 
U.S.  and  European  design  practices,  business 
metnods,  etc. 

He  participated  in  the  1979  Defense  Science 
Board  Summer  Study  on  Reducing  the  Unit  Cost  of 
Equipment  and  contributed  to  the  1979  Defense 
Resource  Management  Study.  He  is  a  member  of 
Phi  Beta  Kappa,  where  he  studied  law  at  the 
University  of  California,  Berkeley. 

DR.  ALAN  J.  ROWE 

Dr.  Rowe  currently  is  a  Professor  of  Management 
and  Policy  Sciences  in  the  School  of  Business 
Administration  at  U.S.C.  He  was  previously  Dir¬ 
ector  of  the  Decision  Systems  Program,  Associ¬ 
ate  Dean  and  Chairman  of  the  Management  Depart¬ 
ment.  He  was  responsible  for  programs  in  Den¬ 
mark,  Tehran  and  Singapore  covering  Management 
Decision  Making,  Management  Control  Systems, 
Management  Information  Systems  Project  Manage¬ 
ment,  and  Acquisition  Management. 

Prior  to  joining  the  U.S.C.  Faculty  in  1965, 

Dr.  Rowe  was  the  Director  of  Industrial 
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Dynamics  on  the  Corporate  Staff  of  the  Hughes 
Aircraft  Company.  He  is  previously  manager  of 
Management  Controls  on  the  Corporate  Staff  of 
System  Development  Corporation,  and  Corporate 
Consultant  in  Manufacturing  and  Computer 
Systems  at  General  Electric. 

His  academic  Career  has  included  teaching  at 
UCLA,  Syracuse,  Columbia  and  NYU  in  addition 
to  USC.  He  is  active  in  professional  socie¬ 
ties  and  is  listed  in  American  Men  of  Science 
and  Who’s  Who  in  the  World. 

He  received  an  Interdisciplinary  Ph.D.  in  Eng¬ 
ineering  from  UCLA  and  his  BS  and  MS  from  Co¬ 
lumbia  University. 

DR.  HEROLD  A.  SHERMAN 

Dr.  Herold  A.  Sherman,  Associate  Professor  of 
Systems  Management,  served  as  Executive  Dir¬ 
ector  of  the  University  of  Southern  California 
Institute  of  Safety  and  Systems  Management 
from  1968  to  1980. 

During  those  ]2  years  the  Institute  grew  from 
a  small  division  of  USC’s  University  College 
to  a  major  element  of  USC  with  over  60  centers 
in  the  United  States,  the  Pacific,  the  Far 
East,  and  West  Germany. 

As  Executive  Director  Dr.  Sherman  administer¬ 
ed  three  USC  degree  Programs.  The  Institute 
baccalaureate  and  master’s  degree  programs  in 
Safety  were  among  the  first  developed  at  major 
universities.  The  innovative  master’s  degree 
program  in  Systems  Management,  with  a  cur¬ 
rent  international  enrollment  of  over  2000, 
has  become  a  model  for  off-campus  degree  pro¬ 
grams  in  the  United  States. 

Dr.  Sherman  was  also  responsible  for  the  over¬ 
all  administration  of  major  research  contracts 
(funded  at  over  $5,000,000)  conducted  by  the 
Institute  for  several  federal  agencies.  The 
Institute  activities  managed  by  Dr.  Sherman 
further  included  an  extensive  program  of  non¬ 
degree  extension  courses  in  Safety. 

Dr.  Sherman  formerly  served  as  President  of 
the  Flight  Safety  Foundation,  an  international 
non-profit  organization.  During  his  term  as 
President  he  reorganized  and  revitalized  the 
Foundation's  programs,  bringing  it  into  a 
sound  fiscal  position.  The  Flight  Safety 
Foundation’s  membership  now  includes  most  of 
the  major  air  carriers  of  the  world  and  the 
majority  of  aerospace  manufacturers. 

Dr.  Sherman,  who  has  been  published  in  the 
field  of  aviation  product  liability,  main¬ 
tains  a  broad  spectrum  of  professional  acti¬ 
vities.  He  is  a  member  of  numerous  pro¬ 
fessional  societies,  has  testified  before 
Congress  concerning  aviation,  and  has  been  an 
invited  speaker  at  both  foreign  and  domestic 
universities  and  corporations.  He  is  a  for¬ 
mer  winner  of  the  Air  Force  Association’s 
Outstanding  Educator  Award  and  the  Strategic 
Air  Command's  Award  for  Outstanding  Achieve¬ 
ment  in  Aerospace  Education.  Dr.  Sherman  is  a 


member  of  the  Transportation  Research  Board  of 
the  National  Research  Council,  National  Academy 
of  Sciences. 

Dr.  Sherman  was  awarded  his  Doctorate  in  Busi¬ 
ness  Administration  by  the  University  of  South¬ 
ern  California  where  he  has  been  a  faculty 
member  since  1965. 

Primary  areas  of  research  interest  include  con¬ 
tract  management  and  procurement  management. 

MR.  J.  RANDOLPH  SIMPSON 
(Alternate  for  Dr.  Varley) 

Mr.  Simpson  is  a  Scientific  Officer  in  the  Op¬ 
erations  Research  Branch  of  the  Mathematic's 
Division  of  the  Office  of  Naval  Research  in 
Arlington,  Virginia.  His  background  is  heavy 
in  economics.  He  has  worked  for  the  U.S.  Navy 
since  1952  in  Logistics  Research,  Naval  Analy¬ 
sis  (anti-submarine  warfare) ,  and  Decision 
Science.  His  current  research  efforts  include 
Risk  Analysis,  Statistical  Decision  Theory, 
Simulation,  and  Optima]  Design  of  Information 
Systems . 

DR.  JOHN  STANHAGEN 

Dr.  John  Stanhagen  is  currently  the  Director 
of  Product  Support  Advanced  Development  at  the 
Douglas  Aircraft  Company.  He  holds  a  Masters 
Degree  in  Logistics  Management  from  the  Air 
Force  Institute  of  Technology  and  is  a  Doctor 
of  Business  Administration  and  Industrial  Man¬ 
agement  (University  of  Southern  California). 
Also,  he  has  served  as  Chairman  of  the  Systems 
Management  Department  at  the  University  of 
Southern  California.  Dr.  Stanhagen  is  a  Fellow 
of  the  Society  of  Logistics  Engineers. 

DR.  ROBERT  M.  STARK 

Robert  M.  Stark  is  Professor  of  Mathematical 
Sciences  and  Civil  Engineering  and  Chairman  of 
the  Graduate  Program  in  Operations  Research 
at  the  University  of  Delaware.  His  research, 
teaching,  and  consulting  for  nearly  two  decades 
has  related  to  applied  probability  and  optimi¬ 
zation  in  the  planning,  design,  and  operation 
of  civil  engineering  related  projects. 

DR.  CHARLES  W.  N.  THOMPSON 

Dr.  Charles  W.  N.  Thompson  is  presently  a  Pro¬ 
fessor  of  Industrial  Engineering  and  Manage¬ 
ment  Sciences,  The  Technological  Institute, 
Northwestern  University.  He  received  his  BS 
from  Kutztown  State  College,  an  LL.B,  Harvard 
Law  School,  an  MBA,  Ohio  State  University  and 
a  Ph.D.  Northwestern  University. 

He  has  had  14  years  teaching  at  Northwestern 
University,  6  years  at  Admiral  Corporation, 
Chicago,  and  6  years  at  Wright-Pat terson  AFB. 

He  was  Chief,  Electronic  Reconnaissance  Sec¬ 
tion,  Aerial  Reconnaissance  Laboratory  at  WPAFB, 
responsible  for  design  and  management. 
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including  the  first  two  major  systems  design¬ 
ed  and  procured  as  complete  airborne  reconnai¬ 
ssance  systems  with  ground  support  components. 
Developed  a  systems  procurement  of  a  major  com¬ 
puter  based  data  handling  system. 

At  Admiral  Corporation,  he  was  Director  of  Eng¬ 
ineering  Services,  which  included  systems  and 
advanced  engineering.  Responsible  for  and 
participated  in  design  and  management  of  maj¬ 
or  en  route  air  traffic  control  bright  dis¬ 
play  system,  and  in  systems  design  over  a 
variety  of  battlefield  sensor  and  communi¬ 
cation  equipments. 

Various  information  and  decision  system  de¬ 
signs  and  resigns,  including  two  political 
caucus  decision  processes,  and  a  quasi-ju¬ 
dicial  university  hearing  and  appeals  system. 

DR.  THOMAS  C.  VARLEY 

Dr.  Varley  has  been  Director  of  the  Opera¬ 
tions  Research  Program,  Office  of  Naval  Re¬ 
search,  Arlington,  Virginia,  since  1970.  He 
is  a  Doctor  of  Business  anc*  Economic  Statis¬ 
tics  from  George  Washington  University,  Wash¬ 
ington,  D.C.  As  a  member  of  the  Operations 
Research  Society  of  \nerica  he  has  held  many 
positions,  including  chairman  in  1976  and 
1977.  Dr.  Varley  has  also  functioned  as 
chairmaa  for  the  Institute  of  Management 
Sciences  College  on  Logistics  (1975-1980)  and 
the  DoD  Ninth  Annual  Acquisition  Symposium 
(1979).  He  is  currently  a  member  of  the 
Board  of  Editors  for  t  irt  Naval  Logistics 
Research  Quarterly. 

MR.  JAMES  WILDER 

Mr.  Wilder  is  Manager  of  the  Econometrics 
Section  in  the  Operations  Analysis  Depart¬ 
ment  of  Grumman  Aerospace  Corp.  This  sec¬ 
tion  engages  in  the  parametric  costing  of 
advirced  aerospace  systems,  and  other 
economic  analyses.  He  has  been  involved 
in  the  development  of  various  methodologies, 
particularly  project  evaluation  and  cost 
risk.  He  earned  his  undergraduate  degree 
in  Mechanical  Engineering  from  Cornell, 
and  he  holds  master:'  degrees  in  Manage¬ 
ment  Engineering  from  Long  Island  Univ¬ 
ersity,  and  in  Applied  Mathematics  from 
SUNY,  Stony  Brook. 

DR.  GEORGE  H.  WORM 

Dr.  George  H.  Worm  is  currently  Associate 
Professor  of  Industrial  Management  at  Clem- 
son  University.  He  has  a  Ph.D.  in  Manage¬ 
ment  Science  from  the  University  of  Tenn¬ 
essee,  and  M.S.  in  Mathematics  from  Florida 
State  University  and  a  B.S.  in  Mathematics 
from  Louisiana  Tech.  He  has  researched  and 
published  in  Mathematics,  Statistics  and 
Management  Science.  His  consulting  ex¬ 
perience  includes  AFSC,  Emory  Medical 
School,  Corp.  of  Engineers,  NASA  and  others. 


He  is  a  member  of  Sigma  XI,  ORSA/TIMS  and 
Beta  Gamma  Sigma. 

MR.  ROBERT  BLACK 

Mr.  Black  is  a  Senior  Analyst  in  the  Econo¬ 
metrics  Section  of  the  Operations  Analysis 
Department,  Grumman  Aerospace  Corp.  He  has 
been  actively  involved  in  methodology  develop¬ 
ment,  as  well  as  performing  analytical  and 
statistical  Industrial  Engineering  studies. 

His  undergraduate  degree  is  B.S.  in  Physics 
from  Hofstra  University,  and  he  holds  masters 
degrees  in  Mathematics  from  Adelphi,  and 
Operations  Research  from  Polytechnic  Institute 
of  Brooklyn. 

DR.  JOHN  R.  ADAMS 

Dr.  John  R.  Adams  holds  the  Ph.D.  in  Business 
Administration  from  Syracuse  University,  and 
is  currently  an  Associate  Professor  or  Organi¬ 
zation  and  Management  at  Western  Carolina  Uni¬ 
versity,  North  Carolina.  He  is  also  the  Dir¬ 
ector  for  Educational  Services  of  the  Project 
Management  Institute.  Dr.  Adams  has  published 
a  number  of  articles  on  various  aspects  of 
project  management,  risk  and  uncertainty  anal¬ 
yses,  weapon  systems  acquisition,  and  logis¬ 
tics  management  in  nationally  distributed 
journals,  and  is  a  frequent  speaker  at  nation¬ 
al  professional  and  at  Department  of  Defense 
symposia.  His  experience  include  management 
of  major  weapon  systems  acquisition  programs 
and  supervision  of  a  major  Air  Force  research 
laboratory . 

PARTICIPANTS 
Name  &  Organization 

Abeyta,  Richard 

Army  Logistics  Management  Center 
Fort  Lee,  VA 

Adams,  John  R. 

Western  Carolina  University 
Cullowhee,  NC 

Alcalay,  J.A. 

Hughes  Aircraft  Company 
Culver  City,  CA 

Beverly,  John 
University  of  Notre  Dame 
Notre  Dame,  IN 

Busch,  Jerry 
P.0.  Box  551 
Burbank,  CA 

Busch,  Jo  Karen 

Western  Carolina  University 

Cullowhee,  NC 
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Burt,  David 

San  Jose  State  University 
San  Jose,  CA 

Cozzol  ino  ,  John  M. 

University  of  Pennsylvania 
Phil adel phi s ,  PA 

DeMong,  Richard  F . 

Air  Force  Business  Research 
Management  Center 
Wright-Patterson  AFB ,  OH 

Demmy,  W.  Steven 
Wright  State  Universi ty 
Dayton,  OH 

Dorfman,  Merlin 

Lockheed  Missiles  and  Space  Corp. 
Sunnyvale,  CA 

Douglas,  James 
McDonnell-Douglas 
Long  Beach,  CA 

Douglass,  J.W. 

Cornell  University 
Ithaca,  NY 

Drake,  James  F. 

Hughes  Aircraft  Company 
Culver  City,  CA 

Edwards,  Ward 

University  of  Southern  California 
Los  Angel  es ,  CA 

El  Shaieb,  Abdul 

San  Jose  State  University 

San  Jose,  CA 

Fargher,  John 

Defense  Systems  Management  College 
Fort  Belvoir,  VA 

Feiler,  Fred 
LOG/AN,  Inc. 

Los  Angeles,  CA 

Fraker,  John 
University  of  Dayton 
Dayton,  OH 


Golden,  Robert  F. 

Air  Force  Business  Research 
Management  Center 
Wright-Patterson  AFB,  OH 

Goodman,  Dick 

University  of  California  at  Los  Angeles 
Los  Angeles,  CA 

Ireland,  Lewis  R. 

SWL ,  Inc. 

McLean,  VA 


Jones,  George 

University  of  Southern  California 
Los  Angeles,  CA 

Keller,  A.Z. 

University  of  Bradford 
Bradford,  West  Yorkshire 

Kennedy,  John  J. 

University  of  Notre  Dame 
Notre  Dame,  IN 

Kirkwood,  Craig  W. 

Woodward-Cl yde  Consultants 
San  Francisco,  CA 

Klopp,  Jerry  A. 

Army  Procurement  Research  Office 
Fort  Lee,  VA 

Kraemer,  George 
Boeing  Vertol  Company 
Philadelphia,  PA 

Kratochvil  ,  Frank 

Grumman  Aerospace  Corporation 

Bethpage,  NY 

Lamm,  Dave 

Naval  Postgraduate  School 
Monterey,  CA 

Larew,  Richard 
Ohio  State  University 
Columbus,  OH 

Launer,  Robert  L. 

Army  Research  Office 
Durham,  NC 

Lorette,  Richard 
Ball  State  University 
Muncie,  IN 

Martin,  Martin  D. 

Air  Force  Business  Research 
Management  Center 
Wright-Patterson  AFB,  OH 

Maust,  Gregory 
Doty  Associates,  Inc. 

Rockville,  MD 

McChesney,  Jack  L. 

Air  Force  Institute  of  Technology 
Wright-Patterson  AFB,  OH 

McLaughlin,  Linda 
Air  Force  Business  Research 
Management  Center 
Wright-Patterson  AFB,  OH 

McNichols,  Gerald 

Management  Consultants  and  Research,  Inc. 
Fal  1  s  Church,  VA 
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Monczka  ,  Robert  M . 

Michigan  State  University 
East  Lansing,  MI 

Moolin,  Frank 

Alaska  International  Industries 
Anchorage,  AK 

Moskowitz,  Herbert 
Purdue  University 
West  Lafayette,  IN 

Rich,  Michael 

The  RAND  Corporation 

Santa  Monica,  CA 

Rippy,  Doug 
University  of  Dayton 
Dayton,  OH 

Rowe,  Alan  J. 

University  of  Southern  California 
Los  Angel  es ,  CA 

Sherman,  Herold 

University  of  Southern  California 
Los  Angeles,  CA 

Sherman,  Stanley 

The  George  Washington  University 
Washington,  DC 

Simpson,  Randy 

Office  of  Naval  Research 

Arlington,  VA 

Solem,  Robert  J. 

Motorola,  Inc. 

Scottsdale,  AZ 

Somers,  Ivan  A, 

Hughes  Aircraft  Company 
Los  Angeles,  CA 

Stanhagen,  John 
Douglas  Aircraft  Company 
Long  Beach,  CA 

Stark,  Robert 
University  of  Delaware 
Newark,  DE 

Tankersley,  Michael  C. 

Air  Force  Business  Research 

Management  Center 

Wri ght-Patterson  AFB.  OH 

Terry,  Robert 
University  of  Utah 
Salt  Lake  City,  UT 

Thompson,  Charles  W.N. 

Applied  Science  and  Technology  Associates, 
I  nc . 

Evanston,  IL 


Thurman,  William 

Defense  Systems  Management  College 
Fort  Bel  voir,  VA 

Varley,  Thomas 

Office  of  Naval  Research 

Arlington,  Va 

Whittington,  Wayne 
NASA  JSC 
Houston,  TX 

Wilder,  James  J. 

Grumman  Aerospace  Corporation 
Bethpage,  NY 

Worm,  George 
Clemson  University 
Clemson,  SC 

Yager,  Ron 
Iona  College 
New  Rochelle,  NY 
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MANAGING  RISKS  AND  UNCERTAINTY : 

LOOKING  TO  THE  FUTURE 
General  William  Thurman,  Commandant 
DSMC  Fort  Belvoir,  VA  22060 

I  appreciate  the  opportunity  to  address  this 
august,  multidisciplinary  group.  And  I  look 
forward  with  great  anticipation  to  the  end 
products  of  this  conference  as  they  may  af¬ 
fect  and  assist  us  in  our  complicated  world 
of  project  management.  (See  page  13 

Highly  developed,  modem,  and  technologically 
based  societies  look  to  the  future.  Likewise, 
change,  progress,  and  planning  are  integral 
parts  of  our  defense  structure,  and  are  cru¬ 
cial  to  our  management  processes.  We  in 
government  often  emphasize  a  future  that  will 
be  characterized  by  complex  weapons  systems. 
These  systems  will  create  management  problems 
and  opportunities  that  far  surpass  our  cur¬ 
rent  contemporary  management  theory  and 
practice.  (See  page  13) 

Formal  recognition  and  emphasis  on  risk  and 
uncertainty  analysis  in  DOD  resulted  from  a 
31  JiO  69  memorandum  from  David  Packard,  DEP 
SEC  DEF  to  the  MIL  DEPs  identifying  problem 
areas  in  the  weapon  system  acquisition  pro¬ 
cess.  In  this  memo,  Secretary  Packard  cited 
inadequate  identification  and  consideration 
of  risks  in  major  programs  as  a  problem  area 
requiring  action  and  that  risk  analysis  be 
applied  by  program  managers  in  their  daily 
activities.  This  growing  recognition  of  the  • 
need  for  risk  and  uncertainty  analysis  over 
the  past  decade  has  been  attended  by  the  need 
for  philosophical  foundations.  This  need  has 
been  manifest  in  many  ways.  Numerous  R&D  or¬ 
ganizations  have  reorganized  in  attempts  to 
produce  more  efficient  systems  of  greater  com¬ 
plexity.  Many  universities  have  revised  theie 
programs  to  provide  multidisciplinary  educa¬ 
tional  opportunities  for  personnel  with  train¬ 
ing  in  engineering  and  science,  as  well  as 
from  the  more  traditional  academic  back¬ 
grounds.  The  past  decade  has  also  witnessed  * 
an  explosion  of  books  and  technical  papers  ) 
on  risk  and  uncertainty  analysis.  The  PM  has' 
recognized  the  benefits  of  this  analysis  and 
the  required  resultant  management  of  these 
risks.  It  is  well  known  that  the  PM’s  task 
is  the  management  of  the  project's  risk  and 
uncertainties.  It  is  evident  that  this  de¬ 
cade  will  require  the  DOD  to  come  to  terms 
with  an  imposing  array  of  problems:  afford¬ 
ability  of  complex  weapons  systems,  suffi¬ 
cient  manpower  to  operate  and  maintain  this 
equipment,  and  the  ever  increasing  threat 
posed  by  the  Warsaw  Pact  as  it  affects  the 
world  balance  of  power.  Tremendous  risks 
and  attendant  uncertainties  must  be  taken  if 
we  are  to  progress.  Technology  must  be  chal¬ 
lenged  in  many  areas  if  we  are  to  overcome 
the  numerical  superiority  of  the  Warsaw  Pact. 


The  common  element  to  many  complex  programs 
is  uncertain  technology.  Whether  we  talk 
about  new  construction  methods,  electronic  de¬ 
vices,  aircraft,  or  tanks,  there  are  numerous 
uncertainties  that  derive  from  inadequate 
knowledge  of  the  basic  technology  or  its 
specific  implementation.  These  problems  have 
no  clear  cut  solutions.  Largely  because  of 
the  number  of  organizations  involved,  as  well 
as  the  uncertainties  and  risks,  the  problems 
and  processes  involved  can  be  typified  as 
lacking  of  boundary  conditions;  solutions 
must  be  derived  on  the  spot,  not  simply  from 
scientific  and  engineering  principles;  and 
implementation  is  going  to  involve  collabora¬ 
tive  and  integrated  activities  that  criss¬ 
cross  organizational  and  national  boundaries. 
As  has  been  evident  in  the  private  sector, 
it  would  appear  that  an  increasing  number  of 
defense  ventures,  because  of  their  size,  capi¬ 
tal  requirements,  and  technological  uncertain¬ 
ty,  will  require  collaborative  "federations" 
as  well.  When  the  countries  within  NATO  re¬ 
cently  sought  to  exploit  new  technologies  for 
early  warning  of  an  attack  by  Warsaw  Pact 
forces,  the  sovereign  countries  joined  forces 
to  coproduce  18  NATO  Airborne  Early  Warning 
Systems.  (See  page  13) 

Traditionally,  ours  is  an  optimistic  society. 
American  technological  pragmatism  is  un¬ 
challenged.  Confidence  in  the  expertness  with 
which  we  can  move  from  research  through  de¬ 
velopment  to  production,  even  when  the  un¬ 
certainties  and  risks  are  staggering,  has  re¬ 
sulted  in  a  growing  demand  for  engineering- 
management  "know  how"  being  applied  to  solve 
any  problem.  The  transferability  of  techno¬ 
logy  and  our  omnipotence  to  solve  any  problem 
need  to  be  tempered.  There  is  an  impressive 
and  growing  list  of  failures  in  large-scale 
advanced-technology  programs.  Many  of  these 
failures  involve  military  programs,  but  there 
have  also  been  numerous  failures  in  non-mili¬ 
tary  projects.  We  know  less  than  we  think  we 
do  about  the  management  process  by  which  new 

j  technology  is  converted  into  operating  systems. 
(See  page  14) 

!  A  major  paradox  within  defense  is  that  effect¬ 
iveness  in  development  programs  has  been  shown 
to  require  a  high  order  of  responsible  auto¬ 
nomy  and  the  opportunity  to  innovate  and  even 
change  plans.  These  complex  projects  with 
very  demanding  performance  requirements  and 
cost  and  schedule  constraints  also  require* 
precise  integration  and  coordination.  While 
parts  of  a  total  system  (i.e.  engine,  air¬ 
frame,  armaments  and  their  sub-assemblies)  are 
designed  and  fabricated  at  distant  places  by 
separate  industrial  firms,  they  must  interface 
perfectly  not  only  in  form,  fit  and  function 
but  within  schedule  &  budgetry  constraints. 

(See  page  14) 

Even  planning  is  a  rather  different  function 
in  developmental  systems  where  risk  and  un- 
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certainty  predominate.  Traditionally,  mana¬ 
gers  are  taught  to  identify  their  ultimate 
need,  to  set  objectives  that  will  help  them 
obtain  the  required  operational  capability, 
and  then  develop  their  acquisition  strategy 
and  plans.  Unfortunately,  this  logical  se¬ 
quence  is  inverted  in  the  real  world. 

Planning  for  uncertainty  and  risk  assessment 
turns  nut  to  be  a  dynamic,  iterative  process. 

In  traditional  management  theory,  managers 
are  expected  to  collect  and  weigh  facts  and 
probabilities,  make  an  optimal  decision  and 
see  that  it  is  carried  out.  In  developmental 
projects,  a  clear  sequence  is  not  possible  be¬ 
cause  of  the  extended  duration,  the  many  tech¬ 
nical  unknowns,  the  continual  discovery  of  new 
"facts'**  and  the  changing  constraints  and  pres¬ 
sures.  The  process  must  be  designed  that  al¬ 
lows  recommitment,  reassessment  and  redirec¬ 
tion  without  allowing  never-ending  improve¬ 
ment  and  excuses  for  missing  budgets  and 
schedules . 

In  the  area  of  contracting,  traditional  con¬ 
tract  monitoring  is  perceived  as  a  legal  and 
accounting  problem.  The  whole  concept  of  res¬ 
ponsibility  changes,  however,  when  the  designs 
keep  evolving,  risks  are  undefined  and  sub¬ 
stantial  uncertainty  is  involved.  There  be¬ 
comes  no  clear-cut  division  between  the  plan¬ 
ners,  the  doers  and  the  monitors.  The  custo¬ 
mer,  DOD,  must  be  able  to  speak  and  under¬ 
stand  the  language  of  the  contractor's  ex¬ 
perts,  to  know  as  much  about  the  problems  and 
risk  as  the  contractors,  to  check  and  supple¬ 
ment  the  work  of  the  government  laboratories 
and  contractors,  and  to  step  in  when  required 
with  the  necessary  specialists  to  help  solve 
the  problem.  I 

Modern  development  programs  have  life  hist¬ 
ories  filled  with  unanticipated  crises,  un¬ 
predicted  barriers,  and  impediments.  What 
appears  to  be  a  reasonable  design,  given  prior 
knowledge  and  experience,  turns  out  to  have  a 
neglected  small,  crucial  factor,  and  some  sub¬ 
part — say  a  new  valve,  fails  to  work  properly. 
This  in  turn  means  that  the  subsystem  may  have 
to  be  redesigned  to  "work  around"  the  problem, 
which  in  turn  affects  other  subsystems  and  ul¬ 
timately  the  system.  A  small  technical  pro-  1 
blero  thus  pyramids  within  an  interdependent, 
integrated  system. 

Obsolescence  is  a  continuing  threat.  The 
mission  requirements  accepted  when  the  concept 
was  chosen  may  be  obsolete  by  the  time  the 
program  is  nearing  the  production  decision. 
Newer  technologies  or  new  data  may  make  the 
chosen  alternative  wasteful,  or  unable  to  meet 
the  new  threat. 

Recognizing  this  obsolescence  factor,  partici¬ 
pants  in  a  project  may  develop  changes  to  in¬ 


corporate  the  most  up-to-date  technology. 

There  is  a  tendency  for  schedule  slippages  to 
be  compounded  as  professionals  in  the  program 
seek  to  improve  the  design — this  further  de¬ 
lays  the  project,  increasing  costs,  &  compli¬ 
cating  development  process. 

By  definition,  complex  programs  are  mission- 
oriented  and  require  an  interdisciplinary  ef¬ 
fort.  Most  professionals,  however,  have  been 
trained  and  are  experienced  in  the  context  of 
specialization,  and  they  expect  to  pursue 
careers  in  that  specialty.  The  welding  and 
interweaving  of  professionals,  and  the  pro¬ 
fessional  knowledge  required  for  risk  assess¬ 
ment  and  planning  for  uncertainty,  challenges 
the  professional  and  manager  alike.  They  must 
find  ways  to  redirect  the  areas  of  interest 
into  risk  and  uncertainty  fields,  while  pro¬ 
viding  for  maintenance  of  the  ties  to  the 
professional’s  field  of  expertise. 

The  DOD  Acquisition  Process  is  in  actuality  a 
phased  project  planning  system.  During  the 
early  stages,  simultaneous  parallel  efforts 
are  encouraged.  This  technique  is  adopted  to 
reduce  the  risk  from  uncertain  technology  de¬ 
velopment  and  to  determine  objectives  and 
their  feasibility,  as  well  as  the  potential 
cost,  scheduling,  performance  &  management  di¬ 
mensions  of  the  program.  If  these  exploratory 
studies  are  not  made,  a  contractor  may  be  en¬ 
gaged  to  start  costly  development  work  on  a 
project  where  the  risks  remain  unknowns.  Ex¬ 
penses  mount  up  rapidly  in  such  cases.  For 
these  reasons,  the  management  system  adopted 
provides  for  a  series  of  go-no  go  decision 
points.  Management  can  halt  the  program  at 
three  points  during  the  process  even  after  ap¬ 
proving  the  MENS.  In  reality,  there  are  no 
automatic  route  markers,  each  program  actually 
uses  a  "tailored"  approach.  (See  page  16) 

Developing  parallel  efforts,  as  described  in 
OMB  Circular  A-109,  stems  from  the  need  to  ex¬ 
plore  competing  concepts  and  approaches  to  op¬ 
timize  life-cycle  cost,  schedule  &  performance, 
taking  into  account  known  risks  and  the  possi¬ 
bilities  for  uncertainties.  Competition  is 
induced  to  take  advantage  of  the  technical  and 
cost  motivation  to  the  contractors  created  in 
this  environment,  and  to  prevent  an  advocacy 
for  a  technology  which  may  yet  be  proven.  Fol¬ 
lowed  literally,  these  parallel  efforts  would 
be  developed  at  each  phase  of  the  acquisition 
cycle,  to  the  extent  that  substantial  risks 
and  uncertainty  exist,  and  that  it  is  economi¬ 
cally  feasible. 

Now  let  us  turn  to  the  specific  areas  of  risk 
and  uncertainty  analysis.  DSMC  has  developed 
a  taxonomy  for  the  area  of  operations  research 
/systems  analysis.  Under  decision  analysis, 
we  find  the  area  of  risk  analysis  which  also 
includes  uncertainty  analysis.  Our  defini¬ 
tion  for  decision  analysis,  decision  risk 
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analysis,  sensitivity  analysis,  and  uncertain¬ 
ty  analysis  are  devined  as  shown  here.  These 
analyses  techniques  are  employed  to  discrimi¬ 
nate  between  parameters  or  a  range  of  para¬ 
meters  which  would  cause  an  alternate  strategy 
to  be  selected.  The  purpose  for  the  use  of 
these  techniques  is  to  improve  the  decision¬ 
maker's  decision  by  providing  a  structure  for 
the  analysis  and  quantifying  the  differences 
between  parameters,  to  provide  insight  and 
understanding  into  the  decision,  and  hopefully 
to  involve  the  decisionmaker  in  the  analysis, 
even  if  only  on  a  "what  if"  basis. 

(see  page  18) 

Risk  has  two  sides,  the  probability  of  failure 
and  the  consequence  of  failure.  Uncertainty 
also  has  similar  considerations  in  determining 
the  probabilities  associated  with  potential 
outcomes  and  also  the  consequence  of  these 
outcomes.  The  point  I  want  to  make  is  that 
although  not  all  management  understands  the 
various  risk  and  uncertainty  techniques,  the 
consequences  stated  in  terms  of  performance 
degradation,  cost  increases  and  schedule  slip¬ 
pages  are  well  understood.  A  method  utilized 
at  DS^C  to  evaluate  risk,  uncertainty  and 
their  consequences  is  a  powerful  technique 
using  utility  theory.  Decision  trees  are  used 
to  map  out  all  possible  relevant  decisions  and 
events  to  define  program  alternatives. 

(See  page  19) 

In  any  analyses,  as  the  risks  and  uncertain¬ 
ties  are  identified,  alternate  approaches  must 
be  developed  and  the  cost,  schedule  and  per¬ 
formance  impacts  considered.  Contingencies 
must  then  be  built  into  the  acquisition  stra¬ 
tegy  to  cover  the  anticipated  cost  escalation, 
correct  possible  performance  shortcomings,  and 
provide  adequate  time  in  the  schedule  to  re¬ 
cover.  (See  page  19) 

The  Reagan  transition  team  has  emphasized  the 
shortcoming  of  the  weapons  research  and  devel¬ 
opment  cycle.  The  Reagan  initiatives  can  be 
expected  to  have  a  tremendous  impact  on  risk 
and  uncertainty  analyses.  The  results  are 
likely  to  be: 

-Less  time  and  raw  data  available  with 
which  to  analyze  risk  and  uncertainty. 

-Greater  use  of  concurrency  and  acceptance 
of  increased  risks  and  uncertainty  at  the 
DSARC  decision  points. 

-Shortened  formal  testing  periods  and  re¬ 
duced  test  data. 

-Greater  emphasis  on  off-the-shelf  techno¬ 
logy  and  commercial  products  adapted  for 
military  use  as  well  as  Pre-Planned  Pro¬ 
duct  Improvements  (P3I)  after  fielding 
systems . 

-Increased  use  of  "tailored"  acquisition 
approaches  to  include  a  maturation  phase 
as  opposed  to  a  Full-Scale  Engineering 
Development  (FSED)  phase. 

-Greater  dependence  on  the  PMO's  ability  to 
perform  data  analysis  and  their  own  inde¬ 


pendent  risk  and  uncertainty  analysis  as 
well  as  increased  responsibility  and  auth¬ 
ority  to  deal  with  the  anticipated  prob¬ 
lems  . 

The  above  impacts  will  require  a  greater  em¬ 
phasis  in  the  program  office  to  identify  sche¬ 
dule  risks  and  uncertainty  earlier  in  the  R&D 
cycle  in  order  to  institute  specific  actions 
to  deal  with  the  time-sensitive  factors. 

Early  data  analysis  will,  of  necessity,  be  in¬ 
vestigative  and  actively  incisive  with  the 
emphasis  on  the  discovery  of  the  unexpected. 

Looking  at  the  Acquisition  Life  Cycle,  the 
program  initiation  decision  or  approval  of  the 
MENS  validates  that  a  need  exists  and  approval 
is  given  to  identify  and  explore  alternate  sol¬ 
utions  to  the  mission  need.  During  the  en¬ 
suing  conceptual  phase,  the  PM  is  responsible 
for  developing  his  acquisition  strategy  to 
provide  the  basis  for  selection  of  one  or  more 
systems  for  further  development  to  satisfy  the 
mission  need.  This  acquisition  strategy  in¬ 
tegrates  technical,  business,  and  management 
program  elements.  The  major  risk  and  uncert¬ 
ainty  analyses  areas  include  the  following 
PM  responsibilities: 

-Continuing  analysis  of  assigned  mission 
responsibilities  in  mission  areas. 

-Managing  exploration,  identifying  and 
developing  of  alternative  system  concepts, 
to  meet  mission  element  tasks. 

-assessing  program  risk  areas  and  conduct¬ 
ing  trade-offs  in  performance,  cost  and 
schedule,  to  achieve  best  balance  for 
alternatives . 

-Considering  alternative  maintenance  con¬ 
cepts  to  identify  impact  on  system  design 
and  resources. 

-Preparing  Technology  Assessment  Annex  (TAA) 
to  identify  areas  of  technology  risk  re¬ 
maining  in  the  program. 

-Including  productibili ty  considerations  in 
the  evaluation  of  alternative  design  con¬ 
cepts  to  determine  production  risks,  and 
the  actions  necessary  to  eliminate  those 
risks . 

-Developing  program  management  constraints 
for  selected  program  factors  of  each  al¬ 
ternative  . 

-Developing  resource  projections,  in  terms 
of  program  objectives  and  constraints. 

(See  page  20) 

Because  of  the  lact  of  hard  data  and  uncertain¬ 
ty  in  the  technical  approaches,  this  phase  is  . 
the  most  unstructured,  from  a  risk  &  uncertain¬ 
ty  analysis  standpoint.  Very  few  models  have 
been  developed  that  handle  these  analyses  well. 
Risk  and  uncertainty  analyses  at  this  point, 
however,  are  also  the  most  important  because 
80%  of  the  Life  Cycle  Costs  are  sunk  in  con¬ 
crete  In  this  phase.  After  developing  and 
testing  "breadboard,"  or  experimental  proto¬ 
types  to  prove  out  the  technology  to  reduce  the 
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risk  of  proceeding,  a  DSARC  I  is  normally  held 
on  major  systems  where  alternate  system  design 
concepts  are  approved  for  development.  The 
purpose  of  the  next  phase,  Dem-Val,  is  to  de¬ 
velop  the  alternative  systems  designs  and  est¬ 
ablish  relationships  between  need,  urgency, 
and  risk  and  to  make  practical  trade-offs  in 
system  capability,  cost  and  schedule.  Advanced 
development  prototypes  are  normally  the  vehi¬ 
cle  for  conducting  demonstrations  of  viable  al¬ 
ternative  systems  and  critical  subsystems  in  a 
realistic  environment.  The  acquisition  stra¬ 
tegy  is  expanded  and  refined  by  the  PM.  The 
major  risk  and  uncertainty  analyses  areas  in 
the  Dem-Val  phase  include  the  following  PM  re¬ 
sponsibilities  : 

-Developing  DTC  and  LCC  goals. 

-Completing  the  Demonstration/Validation 
Testing  and  Evaluations,  as  specified  in 
DCP . 

-Continuing  mission  analyses. 

-Conducting  System  Requirements  Review  (SRR) 
and  System  Design  Reviews  (SDR) . 

Preparing  TAA  identifying  areas  of  techno¬ 
logy  risk  remaining  in  the  program. 

(See  page  22) 

After  developing  and  testing  the  AD  prototypes, 
a  DSARC  II  is  held  to  approve  the  system  or 
systems  for  FSED,  including  procurement  of 
long-lead  production  items  and  limited  pro¬ 
duction  of  systems  for  test  and  evaluation. 

The  risks  of  procuring  long-lead  are  evaluated 
at  DSARC  II,  or  a  DSARC  II  is  held  to  weigh  the 
risk  factors.  The  purpose  of  the  last  develop¬ 
ment  phase,  FSED,  is  to  develop,  test,  and 
evaluate  the  total  system  of  all  items  necess¬ 
ary  for  support,  to  include  training.  Pro¬ 
duction  prototypes  are  used  for  operational 
test  and  evaluation.  The  acquisition  strategy 
is  further  expanded  and  refined  for  production 
of  the  weapons  sys*tem,  logistics  support  and 
deployment.  The  major  risk  and  uncertainty 
analyses  areas  ’include  the  following  PM  res¬ 
ponsibilities  : 

-Conducting  Production  Readiness  Review. 

-Continuing  mission  analyses. 

-Developing  funding  requirements  to  support 
production  schedules, 

-Determining  optimal  production  schedules 
and  rates. 

A  DSARC  III  is  held  at  the  end  of  the  FSED 
phase  for  a  production-deployment  decision  to 
ascertain  that  the  mission  element  task  is 
still  valid,  determine  that  the  engineering  is 
complete,  commit  resources  to  product  and  de¬ 
ploy  the  system,  and  evaluate  readiness  to  pro¬ 
duct  the  system.  Various  production  alterna¬ 
tives  to  include  co-production,  leader- follow¬ 
er,  &  second-sourcing  must  be  evaluated  for 
the  DSARC  III. 

Lessons  learned  in  the  acquisition  process  are 
that  the  concept  phase  is  the  critical  phase 
for  risk  and  uncertainty  analysis;  and  that  an 


innovative,  tailored  approach  is  required  for 
every  program.  Although  important  at  every 
phase  and  decision  point,  the  analyses  at  the 
concept  phase  establishes  the  basis  for  the 
rest  of  the  program  by  developing  system  speci¬ 
fications  and  technology  base.  Cost  models  and 
estimates,  threat  models  and  analyses,  new 
technology  and  system  application,  and  concept 
and  trade-off  studies  with  configuration  defi¬ 
nitions  are  initiated  during  this  phase  to  be 
built  upon  in  subsequent  phases. 

It  is  important  to  mention  here  the  four  meth¬ 
ods  that  the  PM  uses  to  reduce  risk  and  uncert¬ 
ainty,  to  be  used  separately  or  in  combination: 

-Test  and  Evaluation. 

-Prototypes  and  Demonstrations. 

-Studies  and  anlyses. 

-Developing  new  ideas  and  concepts. 

(See  page  23) 

The  PM's  choice  is  governed  by  the  stage  of  the 
acquisition  program,  nature  of  the  risk  or  un¬ 
certainty,  and  the  time  and  money  available. 

In  speaking  to  your  first  conference  on  Manag¬ 
ing  Uncertainty  in  Major  Acquisition  Activities, 
my  predeccessor  at  DSMC,  Admiral  Freeman,  off¬ 
ered  some  challenges  to  the  participants. 

There  were: 

-To  set  solid  baselines  and  goals  for  the 
program,  or  the  product  will  not  meet  the 
program  objectives. 

-To  estimate  financial  needs  as  accurately 
as  possible. 

-To  anticipate  the  environmental  catastrophe, 
and  be  prepared  to  deal  with  it. 

(  See  page  23) 

Hope  for  the  best  and  expect  the  worst.  To 
these  challenges  I  would  like  to  idd  some  of  my 
own.  First,  specific  techniques  are  needed  to 
address  the  front  end  of  the  acquisition  where 
data  is  lacking  and  suspect.  More  needs  to  be 
done  on  managing  uncertainty  based  on  the  data 
developed  for  similar  systems.  In  analyzing 
the  critical  elements  in  the  acquisition  pro¬ 
cess,  DSMC  has  constructed  a  matrix  that  is 
the  combination  of  the  experts'  thoughts  on 
which  factors  are  cost  drivers.  Of  signifi¬ 
cance  is  the  high  preponderance  of  high-cost 
drivers.  These  factors  in  acquisition  are  then 
matched  against  the  data  bases  available  to 
measure  them  early  in  the  acquisition  process. 
The  result  is  that  the  high-cost  drivers  have 
poor  data  bases  as  shown  in  orange.  These  high- 
cost  drivers  require  specific  tailored  ap¬ 
proaches  for  risk  and  uncertainty  analysis  be¬ 
cause  of  the  poor  and  sometimes  suspect  data 
base.  (See  page  23) 

In  dealing  with  change,  the  heart  of  the  matter 
in  both  intervention  and  decision-making  is 
quickness  of  response,  the  focusing  collective 
energies  toward  the  solution  of  a  critical  pro¬ 
blem.  The  second  challenge  is  not  only  to  look 
at  models  and  their  interaction  with  the  data 
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for  "out  of  tolerance"  conditions,  but  to  look 
at  trends  in  the  data  to  anticipate  "out  of 
tolerance"  conditions  in  the  future.  Well-pub¬ 
licized  computerized  schedule  control  techni¬ 
ques,  such  as  PERT  networks,  are  often  well  be¬ 
hind  the  action.  Anticipating  trouble  requires 
very  close  contact.  Another  penalty  for  wait¬ 
ing  is  that  in  a  good  many  situations  correct¬ 
ive  action  is  possible  only  during  a  brief 
"window."  CS2  only  shows  cost  imbalances  after 
they  happen.  New  applications  of  existing 
techniques  or  new  techniques  are  required  to 
show  trends  in  risk  and  probabilities  of  un¬ 
certain  outcomes.  The  second  challenge  is  to 
develop  models  that  track  trends  in  the  basic 
given  parameters  of  the  program,  including 
major  objectives,  basic  resources,  and  some  of 
the  basic  restraints  so  that  "surprises"  based 
upon  changes  in  the  requirements  and  rules  or 
"discoveries"  of  changed  parameters  are  identi¬ 
fied  by  the  trend  lines.  These  change  require 
program  changes  all  the  way  down  the  line  in¬ 
cluding  replanning,  renegotiation,  and  re¬ 
analysis  of  many  program  decision,  and  the 
acquisition  strategy.  (See  page  25) 

The  analysts  employed  within  any  program  to 
accomplish  the  risk  and  uncertainty  assessments 
are  normally  also  required  to  accomplish  at 
least  some  of  the  meticulous  planning  and  lay¬ 
ing  out  of  schedules,  costs  and  performance 
criteria.  There  is  sometimes  a  mistaken  belief 
that  a  program  is  nothing  more  than  the  sum 
total  of  a  number  of  these  relatively  routine, 
production-like  functions.  Program  managers 
may  soothe  their  own  doubts,  or  those  of  their 
sponsors,  by  giving  the  impression  that  the 
knowns  far  outweigh  the  unknowns.  If  the  true 
scope  of  the  problems,  and  the  attendant  costs 
were  known,  as  Hirschman  argues,  the  endeavor 
might  never  be  started. 

There  is  a  sharp  contrast  between  the  precision 
of  specifications  and  accounting  in  today's 
and  tomorrow's  high-technology  programs  and 
the  risk  management  process  associated  with 
their  effective  pursuit.  This  risk  management 
is  characterized  by  a  highly  fluid,  iterative 
and  imprecise  series  of  activities  that  re¬ 
quire  a  high  degree  of  personal  interaction 
between  the  analyst  and  the  technical  experts, 
engineers  and  scientists,  consultants,  service 
staff,  contractor  personnel,  and  other  pro¬ 
fessionals  . 

My  third  and  last  challenge  is  to  improve  com¬ 
munications  to  both  the  professional  and  the 
analyst’s  program  manager.  To  do  this,  the 
analyst  must  use  the  language  everyone  in  his 
program  understands:  performance,  cost  & 
schedule.  Analysts,  like  other  professionals 
caught  up  in  the  stringent  demands  of  advanced 
technology,  must  not  only  be  willing  and  able 
to  communicate  with  other  professionals,  but 
they  must  learn  related  parts  of  other  fields. 
Analysts  must  know  a  good  deal  of  engineering, 
contract  management,  quality  assurance,  pro¬ 


duction  control,  logistics,  basis  for  testing, 
elements  of  program  management,  etc.  Risk  and 
uncertainty  analysis  is  a  complicated  process 
of  continuous  interaction  to  keep  the  program 
within  its  constraints  of  time,  money,  and  per¬ 
formance.  It  requires  skills  in  differentia¬ 
tion,  the  evolution  of  change,  and  the  explora¬ 
tion  of  various  alternatives.  It  also  requires 
integration,  and  collaboration  among  the  speci¬ 
alist-designed  subsystems.  Both  integration 
and  differentiation  must  take  place  at  the  same 
time  and  take  the  other  into  account.  Changes 
in  subsystem  components  must  be  made  in  light 
of  the  interaction  between  subsystems. 

A  lot  of  work  remains  in  the  areas  of  manage¬ 
ment  of  risk  assessment  and  uncertainty  fore¬ 
casting.  The  tools  and  techniques,  however, 
cannot  take  the  place  of  the  analyst  and  mana¬ 
ger  who  is  emotionally  capable  of  managing  un¬ 
certainty  in  an  uncertain  environment;  who 
thinks  logically  and  in  terms  of  alternatives, 
identification  of  problems  and  back-up  stra¬ 
tegies;  and  lastly,  who  has  the  capability  risk 
making  decisions  with  only  limited  or,  some¬ 
times,  suspect  data. 

Thank  you. 

(See  Page  25) 
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FREEMAN'S  CHALLENGES 

•  SOLID  BASELINES  AND  GOALS 

•  FINANCIAL  NEEDS  ACCURATELY  ESTIMATED 

•  ANTICIPATE  THE  ENVIRONMENT 


CRITICAL  ELEMENTS  IN  THE  ACQUISITION 
PROCESS  COMPARISON  MATRIX 
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SECOND  CHALLENGE 

DEVELOP  MODELS  WHICH  TRACK  CHANGES  IN  BASIC  PROGRAM 
PARAMETERS 

•  "OUT  OF  TOLERANCE"  VS  TRENDS 

•  QUICKNESS  OF  RESPONSE 


FINAL  CHALLENGE 

COMMUNICATIONS 

•  COST,  SCHEDULE  &  PERFORMANCE 

•  LEARN  OTHER  FIELDS 
•CONTINUOUS  INTERACTION 
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METHODS  TO  EVALUATE, 

MEASURE  §  PREDICT  COST  OVERRUNS 

A.  J.  Rowe  I.  a.  Somers 

Univ.  of  So.  Calif.  Hughes  Aircraft  Co. 

INTRODUCTION 

Numerous  instances  have  been  documented  of  maj¬ 
or  acquisitions  -  both  military  and  civilian  - 
where  cost  far  exceeded  the  original  estimates. 
With  increasing  levels  of  system  complexities, 
limited  resources,  concurrent  development  and 
production,  constant  changes  in  scope,  conti¬ 
nuous  advance  in  technology  and  urgency  in 
achieving  operational  status,  the  probability 
of  cost  growth  is  becoming  a  critical  aspect 
of  the  acquisition  process.  Even  where  tech- 
nolopy  is  considered  current  state-of-the-art, 
programs  are  not  immune  to  cost  growth. 

Considerable  effort  has  been  expended  in  at¬ 
tempting  to  find  an  answer  to  why  most  acquisi¬ 
tions  have  cost  overruns  and  how  to  control 
their  occurrence.  The  research  reported  here 
builds  on  prior  work  in  the  field  and  provides 
an  approach  for  dealing  with  the  uncertainty 
involved  in  the  acquisition  process.  The  con¬ 
tributions  made  by  E.B.  Cochran  to  the  study  of 
disruption  have  provided  the  foundation  for 
much  of  the  material  covered  in  this  report. 

ACQUISITION  UNCERTAINTY 

)  What  is  now  recognized  is  that  variability  or 
uncertainty  is  an  inherent  aspect  of  the  ac¬ 
quisition  of  major  programs.  The  choice  is  not 
whether  to  incorporate  uncertainty  as  an  inte¬ 
gral  part  of  the  acquisition  process,  but, 
rather  to  recognize  that  "as  contracted  costs," 
which  are  in  effect  single  point  estimates  can 
no  longer  be  used  as  the  basis  for  controlling 
cost  growth. 

The  objective  of  this  research  has  been  to  de¬ 
velop  an  approach  for  predicting  cost  uncer¬ 
tainty,  which  recognizes  that  variability  can¬ 
not  be  eliminated  but  rather  that  there  are 
trade-offs  that  are  available  to  decision  mak¬ 
ers.  The  tradeoffs  are  predicated  on  formu¬ 
lating  cause  and  effect  models  which  provide 
insight  needed  to  improve  the  acquisition  de¬ 
cision  process.  Thus,  inherent  in  the  approach 
presented  here  is  system  dynamics,  interdepend¬ 
encies,  variability,  and  uncertainty  in  the 
acquisition  process.  In  addition,  the  pro¬ 
cesses  used  to  achieve  desired  objectives  are 
also  described.  For  example,  concurrency, 
learning,  change,  technological  advance,  pro¬ 
gram  management,  etc.,  all  are  part  of  the 
acquisition  process  and  contribute  to  the 
effects  observed. 

THE  ACQUISITION  PROCESS 

Any  description  of  the  acquisition  process  is, 
at  best,  only  a  static  representation  of  an  ex¬ 


tremely  complex  set  of  interdependent  activi¬ 
ties.  For  our  purpose,  we  will  use  two  basic 
diagrams  to  aid  in  understanding  the  process. 
The  first,  shown  in  Figure  1  describes  the 
kinds  of  uncertainty  associated  with  acquis 
tions.  (see  next  page) 

The  matrix  shown  in  figure  1  attempts  to  de¬ 
velop  a  basis  for  understanding  causality. 

Thus,  internal  control  assumes  all  things  are 
known  and  controllable  with  estimates  based  on 
past  data,  procedures,  designs,  suppliers,  etc. 
The  other  three  categories,  however,  represent 
the  reality  in  major  acquisition,  even  though 
management  does  not  have  complete  control  or 
ability  to  predict  outcomes.  It  is  this  realm 
of  uncertainty  that  has  significant  impact  on 
cost  and  is  the  principal  emphasis  of  this  re¬ 
port.  The  categories  shown  in  Figure  1  cor¬ 
respond  to  ones  developed  by  the  USAF  Academy 
Risk  Analysis  Study  Team  (129) .  These  are 
shown  on  the  next  page. 


The  description  of  the  USAF  categories  is  as 
follows : 

Internal  Program  Uncertainty:  Deals  with  the 
way  in  which  the  program  is  organized,  planned 
and  managed.  Uncertainty  of  the  initial  esti¬ 
mate  and  its  impact  on  program  management.  Un¬ 
certainty  in  the  acquisition  strategy  and  out¬ 
come.  Uncertainty  in  resources  needed,  flexi¬ 
bility,  or  lack  of  contingency  plans.  Compet¬ 
ing  demands,  including  conflict  between  relia¬ 
bility,  vulnerability  and  maintainability  with 
performance  and  operating  costs. 

Technical  Uncertainty:  Covers  the  feasibility 
of  developing  the  system  at  all,  including  the 
degree  of  technical  difficulty.  Generally 
starts  with  an  optimistic  estimate  of  the  state- 
of-the-art  and  often  leads  to  a  slippery  tech¬ 
nical  baseline. 

Process  Uncertainty:  Deals  with  the  sensitivi¬ 
ty  to  changes  in  the  external  environment  such 
as  changes  in  priorities  or  policies,  the  Presi¬ 
dents  budget,  congressional  political  consider¬ 
ations,  etc.  Unavailability  of  funding/re¬ 
sources  when  needed.  Uncertainty  in  criteria 
used  for  changes,  control,  surveillance,  DSARC 
decisions,  etc.  Effects  of  inflation  and 
government  regulation. 

Target  Uncertainty:  Covers  the  uncertainty  in 
meeting  performance,  cost  or  schedule  goals  and 
determination  of  needs.  Uncertainty  in  trans¬ 
lating  abstract  needs  into  concrete  specifica¬ 
tions.  Problems  of  early  estimates  which  are 
seldom  revised,  (shown  on  page  29) 

A  comparison  of  the  categories  shows  that  there 
is  considerable  similarity  between  the  two. 

The  second  approach  to  understanding  problems 
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FIGURE  1  ACQUISITION  UNCERTAINTY 
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in  Figure  1 

1.  Internal  Control 

2.  Technological  Uncertainty 
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4.  Environmental  Uncertainty 
The  description  of  the  USAF  categories 
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in  Lhe  acquisition  process  and  the  inherent 
cost  overruns  is  represented  by  the  diagram 
shown  in  Figure  2.  Illustrated  are  the  factors, 
the  interdependencies,  and  the  processes  in¬ 
volved  in  acquisition  management.  As  describ¬ 
ed  in  the  USAF  report,  the  acquisition  process 
is  a  tremendously  complex,  turbulent  network 
of  activities  for  which  a  static  model  is  un¬ 
satisfactory  . 

Although  Figure  2  does  not  reflect  the  dynamics 
and  interactions  that  occur  in  an  on-going  or¬ 
ganization,  it  illustrates  a  number  of  key 
concepts  that  will  be  developed  in  the  report. 
The  linkages  between  the  four  basic  uncertain¬ 
ty  variables  and  acquisition  management  help 
to  define  the  processes,  activities  or  vari¬ 
ables  that  contribute  to  the  uncertainty  of 
acquisition  management.  The  four  basic  uncert¬ 
ainty  variables  considered  are: 

1.  Organizational  Slack:  A  measure  of  the 
organization’s  ability  to  perform  the 
task  requirements. 

2.  Customer  Urgency:  The  time  compression, 
concurrency,  or  degree  of  overlap  between 
phase  of  development,  and  change  in 
scope . 

3.  Technological  Uncertainty:  A  measure  of 
the  state-of-the-art  and  the  degree  of 
interdependency  among  system  components. 

4.  Environmental  Uncertainty:  The  factors 
that  cause  disruption,  delays,  shortages, 
failures,  etc.  that  are  not  under  the 
control  of  management  in  the  acquisition 
process . 

The  exterior  linkages  identify  secondary  ef¬ 
fects,  and  relate  the  four  uncertainty  varia¬ 
bles  to  one  another.  Considering  the  whole 
diagram,  one  can  see  that  the  variables  and 
linkages  define  a  network  of  interdependencies 
which  ultimately  contribute  to  the  uncertainty 
and  the  consequent  problems  in  the  acquisition 
process . 

DEFINING  THE  PROBLEM  OF  COST  OVERRUNS 

In  its  report  to  congress,  the  Government  Ac¬ 
counting  Office  (GAO)  (73)  claims  that  major 
weapons’  cost  growth  since  World  War  II  far  ex¬ 
ceeds  the  rate  of  inflation  and  that  various 
efforts  to  restrain  costs  seem  unlikely  to 
achieve  really  substantial  cost  reductions. 

This  statement  by  the  GAO  goes  counter  to  the 
general  belief  that  has  been  held  which  con¬ 
tends  that  closer  scrutiny  of  the  acquisition 
process,  such  as  tighter  control  and  stricter 
acquisition  policies  on  the  part  of  the  govern¬ 
ment,  will  significantly  reduce  the  levels  of 
cost  growth.  However,  control  alone  will  not 
solve  the  problem  since  the  cost  of  additional 
controls  can  be  considered  part  of  a  cost 
growth  -  adding  to  the  problem  rather  than  the 
solution.  Figure  3a  illustrates  the  effects 
of  various  degrees  of  control  on  total  cost. 
Also  shown  is  the  net  benefit  of  control  (value 


less  cost)  as  a  function  of  degree  of  control. 
This  curve  leads  to  an  optimal  value  from  which 
a  change  in  control  (increase  or  decrease)  de¬ 
creases  the  net  benefit.  (See  page  29) 

Figure  3b  illustrates  a  simular  approach  using 
degree  of  planning  as  a  function  of  cost  of 
planning  and  cost  of  errors  (lack  of  complete 
planning.  In  the  case  of  figure  3b,  the  total 
cost  curve  (cost  of  planning  plus  the  cost  of 
errors)  will  have  some  optimal  value  from  which 
a  change  in  degree  of  planning  will  increase 
the  total  cost. 

An  alternative  to  the  use  of  control  as  the 
sole  means  for  reducing  cost  is  an  approach 
based  on  predicting  the  likelihood  of  disrup¬ 
tions  occurring  during  performance  of  a  con¬ 
tract  and  thereby  permit  cost  reductions.  Thus, 
based  on  a  relationship  between  cost  growth 
and  likelihood  of  disruption,  the  control  of 
cost  can  be  improved.  Knowing  when  and  where 
disruptions  might  occur  (with  a  defined  degree 
of  probability),  the  acquisition  manager  can 
develop  a  basis  to  reduce  the  likelihood  of  dis¬ 
ruption  and  thereby  lower  the  level  of  cost 
growth.  In  addition,  other  approaches  can  be 
applied  in  advance  cf  a  probably  disruption 
which  will  also  lower  the  impact  on  cost  growth 
due  to  the  effects  of  disruptions. 

’’Disrupt”  means  to  cause  disorder  or  turmoil. 

In  our  context,  the  term  disruption  refers  to 
the  disorder  and  turmoil  created  in  a  program 
plan  and  in  related  production  procedures, 
which  normally  are  used  to  minimize  costs. 

Hence,  ’’disruption  cost”  is  the  difference  be¬ 
tween  the  actual  cost  for  a  program  on  the  one 
hand,  and  the  cost  "reasonably  required”  to  per¬ 
form  the  task  in  the  configuration  finally  de¬ 
livered,  on  the  other.  The  "should-cos t-as- 
build”  estimate  includes  the  estimated  cost  of 
all  changes  incorporated  in  the  final  deliver¬ 
able  product.  It  does  not  include  penalties 
caused  by  the  late  incorporation  of  changes  (re¬ 
trofit,  rework,  lost  learning,  etc.).  See 
Figure  4.  (Shown  on  Page  29) 

The  existence  of  a  sizable  overrun  always 
raises  the  question  as  to  the  adequacy  of  the 
estimate  used  as  the  basis  to  determine  the 
overrun.  The  overrun  may  reflect  a  poor  assess¬ 
ment  or  errors  in  the  estimating  process,  both 
leading  to  a  low  estimate;  such  a  discrepancy 
itself  could  well  be  disruptive.  We  shall  use 
an  "objective”  definition  of  disruption,  with 
the  should-cost  to  be  based  on  a  realistic  esti¬ 
mate  for  each  organization.  This  suggests  using 
estimating  methods,  which  deal  with  fundamental 
characteristics  of  the  product  such  as  para¬ 
metric  estimating. 

The  Department  of  Defense  (DoD)  and  defense  con¬ 
tractors  have  been  under  increasing  pressures 
to  reduce  the  levels  of  cost  growth  in  the  wea¬ 
pon  systems  acquisition  process.  According  to 
The  Armed  Forces  Journal  (March,  1978)  estimates 
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Weapon  System 
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1.  Dragon  Antitank  Missile 

2.  Condor  Antiship  Missile 

3.  SRAM  Ai r-to-Ground  Missile 

4.  Minuteman  III  Missile 

5.  Main  Battle  Tank 

6.  Heavy  Tank  (in  design) 

7.  Troop  Helicopter 

8.  F -14  Fighter 

9.  F- 15  Fighter 

10.  F-lll  Bomber 

11.  B-l  Bomber 

12.  Nuclear  Attack  Submarine 

13.  Nuclear  Carrier 

14.  Helicopter  Carrier 
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TABLE  2  WEAPON  SYSTEMS  COST  ESTIMATE  COMPARISONS 
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based  on  1977  Government  Accounting  Office  GAO 
data  indicate  the  DoD's  weapon  cost  growth  a- 
verages  less  than  half  of  other  major  federal 
acquisitions.  In  its  Report  to  Congress,  on 
Financial  Status  of  Major  Federal  Acquisitions 

January  1979  (72),  the  GAO  indicates  that  maj¬ 

or  federal  acquisitions  of  all  agencies  are 
estimated  to  cost  $531.2  billion  at  completion. 
This  amount  represents  an  increase  of  $207.4 
billion  or  64%  over  initial  estimates  or  base¬ 
line  estimates  when  initial  estimates  were  not 
available . 

Cost  growth  is  not  limited  to  the  acquisition 
of  weapon  systems  exclusively.  Literature  is 
replete  with  examples  of  cost  growth  on  various 
projects.  Several  examples  are  shown  below 
(94): 
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TABLE  1  EXAMPLES  OF  COST  GROWTH 

When  programs  such  as  the  Apollo  Space  Program 
with  a  200%  overrun,  and  the  Space  Shuttle  Pro¬ 
gram  are  added  to  this  list,  the  types  and  lev¬ 
els  of  technologies  spanned  becomes  diverse. 

It  is  apparent  from  the  examples  cited  above 
that  both  military  and  non-military  projects 
suffer  from  cost  growth.  Furthermore,  that 
cost  growth  is  not  limited  to  programs  that 
have  advanced  state-of-the-art  technology. 

U.S.  News  and  World  Report  its  quotes  a  GAO  re¬ 
port  disclosing  the  cost  of  55  major  weapons 
at  26.3  billion  dollar  above  the  pentagon  esti¬ 
mates  and  still  rising.  Of  $7  billion  increase 
in  the  last  half  of  1973,  it  was  estimated  that 
inflation  accounted  for  $2.5  billion,  engineer¬ 
ing  changes  for  $1.7  billion,  schedule  changes 
for  $1.4  billion  and  other  causes  such  as  re¬ 
search,  development  and  production  difficulties 
accounted  for  the  remaining  $1.4  billion.  Tab¬ 
le  2  below  illustrates  the  per  unit  overruns 
in  a  large  variety  of  weapon  systems  as  re¬ 
ported  in  U.S.  News  and  World  Report  (June 
1974) :  (See  previous  page) 

Major  General  Dewey  Lowe,  at  the  1979  Confer¬ 
ence  on  Managing  Uncertainty  in  the  Acquisition 
of  Major  Programs  reviewed  an  Air  Force  Study 
of  seven  major  aircraft  systems,  including  the 
Bl,  F15 ,  F16,  A10,  E3A,  E4  and  EF111A.  The 
study  revealed  an  improvement  in  the  ability  to 
control  program  cost  growth.  In  the  1950’s 
major  system  cost  growth  exceeded  200%,  while 
systems  suffered  from  high  risk,  poor  defini¬ 
tion  and  low  visibility.  In  the  60’s,  risk  was 


moderate,  definition  was  better,  total  package 
procurement  and  concurrency  was  used,  and  cost 
growth  was  between  100  and  200%.  The  70’ s  have 
witnessed  growth  of  less  than  100%  with  the 
application  of  prototypes,  change  controls  and 
Defense  System  Acquisition  Review  Council.  When 
adjusted  for  inflation,  the  approximately  90% 
growth  rate  was  a  30%  rate  in  base— year  dollars. 
General  Lowe  estimates  that  as  much  as  70-80% 
of  the  cost  overrun  is  caused  by  inflation. 

One  of  the  classic  cases  of  cost  overrun,  the 
Fill,  is  described  in  the  paper  by  Roesch  and 
Sage  (151).  They  report  that  unprecedented 
cost  growth  of  $3  million  per  unit  in  1966  to 
almost  $15  million  per  unit  in  1970  was  caused 
by  ineffective  program  management  on  the  govern¬ 
ment’s  part,  poor  DoD/cont ractor  relationships, 
over-centralization  of  acquisition  management 
in  O.S.D.  and  poorly  defined  operational  re¬ 
quirements.  Here  is  an  illustration  of  the 
kinds  of  problems  that  can  cause  growth  in 
costs . 

As  a  counter-example  to  purely  military  pro¬ 
jects,  Cochran  examined  a  number  of  non-mili¬ 
tary  overruns  as  shown  in  Table  3.  Based  on 
an  analysis  of  the  annual  cost  growth,  which 
ranged  from  4  -  18%,  inflation  accounted  for 
the  major  portion  of  the  overrun  in  6  out  of 
14  programs.  Overall  cost  overruns  ranged  from 
124.3%  to  387%.  The  GAO,  in  its  January  1979 
report  (72),  indicated  that  inflation  accounts 
for  approximately  50%  of  the  cost  growth  in 
military  projects.  Thus,  both  military  and  non¬ 
military  programs  suffer  from  the  effects  of 
inflation.  This  is  becoming  especially  acute 
in  the  era  of  double  digit  inf lation .(next  page) 
(See  page  31) 

Notes  to  TABLE  3 

(1)  Except  for  the  Alaska  Pipeline,  data  is 
derived  from  Table  2  of  report  Alaska 
Natural  Gas  Transportation  System,  U.S. 
Dept,  of  Interior,  July  1,  1977. 

(2)  Includes  changes  in  scope,  effects  of  de¬ 
sign  &  completion  delays  on  time-related 
costs  of  force  majeure  events. 

(3)  Original  estimate  of  $900  million  was 
clearly  just  a  crude  guess.  The  1972 
estimate  of  $2.9  billion  was  based  on 
more  careful  study,  and  is  compared  with 
a  1977  Alyeska  estimate  of  $4.6  billion 
uninflated  cost  to  obtain  the  scope  and 
inflation  breakdown. 

Another  example  of  cost  overruns  as  related  to 
energy  processing  plants  is  shown  in  Figure  5 
(41).  The  overruns  range  from  less  than  200% 
to  almost  700%  for  the  actual  cost  compared  with 
the  initial  estimates . (See  page  32)  ) 

FACTORS  CAUSING  COST  OVERRUNS 

There  have  been  many  studies  attempting  to  iden¬ 
tify  the  causal  factors  involved  in  cost  over¬ 
run.  In  this  section,  we  will  summarize  find¬ 
ings  from  a  number  of  sources  and  use  these 
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Annual  Growth(X)  (2) 
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TABLE  3  -  COMPARISON  OF  RECENT  NON-MILITARY  OVERRUNS 
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Figure  6  -  GAO  CRITICAL  ISSUE  IN  ACQUISITION 
(Report  to  Congress,  GAO,  PSAD-80-43,  June  12,  1980) 
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results,  and  the  cost  trade-off  curves  in  the 
next  section  as  the  basis  for  development  of  a 
model  to  predict  likelihood  of  disruption  and 
the  concommitant  cost  growth. 

The  GAO  and  the  RAND  Corporation  have  done  by 
far  the  most  extensive  and  continuing  analysis 
of  cost  growth.  In  its  report  to  Congress  on 
Impediments  to  Reducing  the  Cost  of  Weapon  Sy¬ 
stems  the  CAO  identifies  the  military's  de¬ 
sire  for  maximum  performance,  high  technology 
weapon  systems  along  with  instability  of  con¬ 
gressional  funding  and  constraints  as  the  maj¬ 
or  contributors  to  cost  growth  (73).  They  also 
identify  low  rates  of  production,  absence  of 
price  competition,  lack  of  motivation  for  con¬ 
tractors  to  reduct  cost,  impact  of  government 
controls,  reduced  R&D  expenditures  and  lowered 
productivity  as  additional  elements  contribu¬ 
ting  to  cost  growth.  Their  conclusions  on  how 
to  reduce  cost  was  to  provide  program  stabili¬ 
ty  that  helps  recover  investments,  put  more 
emphasis  on  invested  capital  than  production 
costs  and  provide  greater  flexibility  in  meet¬ 
ing  changing  priorities  and  needs. 

In  its  June  1980  report  to  the  Congress  (76),  . 
the  GAO  identified  the  issues  that  have  had  a 
direct  impact  on  weapon  system's  mission  ef¬ 
fectiveness.  The  majority  of  the  issues  were 
concerned  with  operational  or  performance  li¬ 
mitations,  survivability  or  vulnerability,  a- 
vailability,  meeting  requirements  and  relia¬ 
bility.  As  concerns  program  acquisition  the  t 
major  issues  dealt  with  system  affordability, 
requirements  of  data  reporting,  concurrency  of 
production  and  development,  inadequate  test¬ 
ing,  cost-effectiveness  assumptions,  lack  of 
qualified  personnel,  indecision  on  system  ur¬ 
gency  and  technical  risks.  A  summary  chart 
of  their  findings  is  shown  in  Figure  6. 

(See  page  32) 

Since  the  late  1950's,  the  RAND  Corporation 
has  conducted  studies  to  describe  the  in¬ 
herent  uncertainty  of  the  product  development 
process.  From  early  1960  to  date  it  has  is¬ 
sued  a  number  of  increasingly  specific  reports 
to  show  that  design  and  production  concurrency 
accompanies  large  overruns,  which  were  avoided 
when  development  work  substantially  preceded 
production.  Quantitative  measures  of  state- 
of-the-art  demonstrated  impressive  correlation 
between  degree  and  the  proportion  of  cost  over¬ 
run  for  cases  of  design  and  production  con¬ 
currency.  As  a  result,  RAND  researchers  re¬ 
commend  reducing  uncertainty  in  advanced  tech-  . 
nology  programs  by  return  to  "incremental" 
product  development  -  a  procedure  which  has 
long  been  used  by  manufactureres  of  commercial 
products  with  stable  design  (162). 

The  RAND  studies  also  show  that  cost  and  sched¬ 
ule  problems  are  larger  as  the  degree  of  state- 
of-the-art  advance  increases.  Sophisticated 
management  planning  and  control  programs  and 
incentive  contracting  have  small  effect,  and 
improvement  through  better  cost  estimating  and 


monitoring  of  cost  growth  does  not  seem  sub¬ 
stantial.  Furthermore,  concurrency  may  not 
even  reduce  development  time  significantly.  In 
short,  the  main  problem  lies  with  the  uncer¬ 
tainty  that  affects  development  and  production 
costs  in  the  presence  of  urgent  time  schedules. 

The  importance  of  uncertainty  as  a  cause  of 
overruns  has  been  documented  by  the  RANI)  stud¬ 
ies,  which  after  over  twenty  years  of  studying 
complex  development  programs  in  the  U.S.  and 
abroad  concluded  that  (146) : 

"High  system  cost  growth  appears  to  arise 
primarily  from  efforts  to  subdue  diffi¬ 
cult  technology  on  highly  compressed 
schedules. . (and  the)  acceptance  of  opti¬ 
mistic  assumptions  about  the  long-term 
predictability  of  technology  and  the  cost 
of  coping  with  it." 

In  describing  system  acquisition  experience. 
Perry,  et.al  (53).,  point  out  initial  esti¬ 
mates  tend  to  be  overly  optimistic  and  do  not 
consider  or  understate  technological  diffi¬ 
culties  actually  encountered  in  program  develop¬ 
ment.  As  a  consequence,  these  difficulties 
which  lead  to  increases  in  total  program  costs 
are  seldom  accounted  for  early  in  a  program. 

They  found  that,  in  nearly  all  cases,  renego¬ 
tiated  contracts  were  much  closer  to  actual 
performance  requirements  and  that  this  was  re¬ 
flected  in  adjusted  costs.  Thus,  the  earlier  a 
prediction  of  cost  is  made,  the  greater  the  ex- 
ipected  uncertainty  of  actual  cost.  In  general, 
in  the  early  conceptualization  stage,  the  re¬ 
quired  technological  advances  and  eventual  sy¬ 
stem  configuration  are  poorly  known.  Their 
conclusions  concerning  cost  growth  and  per¬ 
formance  faults  were  that  they  were  principal¬ 
ly  due  to  changes  in  program  scope  and  they 
were  outside  of  the  contractor’s  control.  The 
factors  outside  of  the  contractor's  control 
generally  accounted  for  the  difference  between 
predicted  cost  of  the  original  program  and  the 
final  cost  of  the  program  as  actually  delivered. 

In  its  attempt  to  control  cost,  the  DoD  insti¬ 
tuted  the  Design  to  Cost  Concept  (DTC)  for  maj¬ 
or  weapon  system  acquisition.  Although  this 
(approach  is  an  attempt  to  keep  cost  within 
limits  that  can  be  achieved  by  a  specified  de¬ 
sign,  it  is  recognized  that  frequent  changes 
i can  undermine  confidence  in  the  process.  Thus, 
flexibility  is  needed  because  of  the  difficulty 
iin  estimating  major  system  costs  with  precision. 
This  implies  that  performance  parameters  must 
be  variable  if  cost  remains  relatively  fixed. 

The  RAND  report  by  Large  (29)  on  bias  in  cost 
estimates  illustrates  another  dimension  of  the 
problem  of  cost  overruns.  In  an  analysis  of 
the  comparison  of  initial  bid  with  final  cost, 
he  found  that  where  the  technological  advance 
required  is  not  fully  understood  the  final  cost 
can  be  off  by  a  factor  of  two.  He  cites  state¬ 
ments  made  by  contractors  which  clearly  in- 
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dicate  that  initial  estimates  are  designed  to 
win  the  competition.  This  approach  to  winning 
the  contract  is  also  called  "buying-in. " 

Martin,  et  al,  (135)  in  their  paper  on  the  re¬ 
lationship  between  cost  and  cost  estimation  in¬ 
dicate  that  a  cost  estimate  is  at  best  a  rea¬ 
soned  guess  about  a  future  outcome.  The  esti¬ 
mate  requires  judgment  and  therefore  is  sub¬ 
jective  in  nature.  Furthermore,  because  cost 
estimates  are  probabilistic  in  nature  and  are 
valid  only  as  long  as  the  assumptions  on  which 
they  were  based  remain  the  same,  there  is  need 
to  revise  estimates  in  consideration  of  the  un¬ 
certainty  which  exists.  They  point  out  that 
the  Jevel  of  uncertainty  is  very  high  in  the 
early  phases  of  an  acquisition,  in  part  due  to 
the  vagueness  of  system  specifications.  The 
uncertainty  is  reduced  as  information  is  ob¬ 
tained  from  testing  and  evaluation.  The  con¬ 
clusion  that  is  drawn  is  that  mistakes  are  made 
using  the  wrong  approach  for  the  given  phase  of 
the  acquisition  cysle.  The  appropriate  cost 
estimating  basis  for  each  phase  is  shown  in 
Figure  7  below. (See  next  page.  Fig.  7  &  Table  ^ 

The  conclusions  drawn  by  Martin,  et  al,  are 
supported  by  studies  which  show  that  initial 
estimates  seldom  reflect  the  final  cost.  Davis 
(12)  supports  this  finding  in  his  analysis  of 
uncertainty  associated  with  cost  estimating. 

He  contends  that  lack  of  information  contri¬ 
butes  to  the  uncertainty  of  cost  estimates.  He 
a3so  maintains  that  diffusion  of  authority  and 
responsibility  for  cost  estimating  throughout 
the  acquisition  cycle  complicates  the  process. 
Furthermore,  short-fuse  requirements  impose 
severe  time  constraints  and  also  contribute  to 
estimating  errors.  He  concludes  that  there  is 
a  need  for  greater  flexibility  in  the  acquisi¬ 
tion  of  major  systems  in  order  to  cope  with 
program  uncertainties.  Single  point  estimates 
do  not  recognize  the  basic  uncertainty  involv¬ 
ed  in  cost  and  cost  range  would  provide  greater 
flexibility.  Basically,  he  argues  that  cost 
overruns  will  be  a  way  of  life  unless  consider¬ 
ation  is  given  to  the  impact  of  program  uncer¬ 
tainty  on  estimating.  To  overcome  this  problem 
the  Army  has  applied  a  cost  estimating  program 
called  TRACE  (Total  Risk  Assessing  of  Cost  Es¬ 
timates).  Although  there  is  no  provision  for 
inflation,  TRACE  does  consider  the  impact  of 
technical  design  changes,  rescheduling,  addi¬ 
tional  testing  and  additional  hardward  require¬ 
ments  (91).  This  approach  specifically  recog¬ 
nizes  that  uncertainty  is  an  inherent  part  of 
the  acquisition  process  and  that  unless  the 
causal  aspects  of  estimating  uncertainty  are 
taken  into  account  cost  overruns  will  continue 
unabated. 

The  problem  of  cost  estimating  is  poignantly 
described  in  the  paper  on  a  theory  of  cost 
growth  (81).  It  describes  cost  as  a  "dependent!* 
variable  that  reflects  action  taken  in  res¬ 
ponse  to  contractual  requirements.  Thus,  if 
cost  becomes  uncoupled  from  requirements,  then 


the  numbers  are  meaningless.  Furthermore, 
point  estimates  speed  up  the  uncoupling.  A 
number  of  reasons  why  costs  become  uncoupled 
from  reality  include: 

1.  Budgets  do  not  anticipate  technical  re¬ 
quirements,  but  instead  react  in  a  lag¬ 
ged  manner  to  changes  in  requirements. 

2.  There  is  a  serious  gap  between  those  who 
are  knowledgeable  about  costs  and  those 
having  technical  knowledge. 

3.  Annual  budgetary  considerations  tend  to 
dominate  the  incremented  acquisition 
process . 

4.  The  sheer  magnitude  of  numbers  of  people 
involved  in  government  functional  areas 
proliferates  the  problem  and  reduces 
flexibility. 

5.  Differential  effects  of  inflation  on 
given  program  components. 

6.  Competition  causes  cost  estimates  to  de¬ 
cline  in  the  pre-contract  award  period. 

7.  Monopsony,  which  is  a  market  condition 
where  the  customer  controls  demand  and 
tries  to  minimize  cost. 

The  recommendation,  then,  is  to  change  these 
practices  so  that  the  acquisition  process  more 
nearly  follows  normal  competitive  practice 
which  would  avoid  cost  growth  based  on  unreal¬ 
istic  initial  bids  and  unreasonable  budgeting 
and  estimating  practices  during  the  acquisition 
cycle . 

In  a  study  conducted  at  USC,  six  major  programs 
were  analyzed  to  determine  the  primary  causes 
for  cost  growth,  schedule  shippage  and  perform¬ 
ance  degradation.  Twenty-six  factors  were 
identified  as  specifically  contributing  to  cost 
overruns.  These  were  placed  into  the  four  ba¬ 
sic  categories  described  in  the  beginning  of 
the  paper.  The  relevance  of  each  factor  to  the 
six  programs  is  shown  in  Table  5.  The  result 
was  that  every  one  of  the  programs  had  cost 
overruns  (see  Table  4)  and  all  encountered 
schedule  slippage  with  BART  and  the  Fill  having 
•performance  degradation.  Furthermore,  customer 
urgency  had  the  most  pervasive  impact  on  all 
programs  with  technological  uncertainty  second, 
organizational  slack  third,  and  environmental 
uncertainty  having  the  smallest  impact.  The 
specific  events  or  factors  contributing  to  the 
effects  shown  in  Table  4  and  5  are  summarized 
in  Table  6.  In  retrospect,  it  is  little  wonder 
that  these  projects,  which  span  civilian  and 
military,  high  and  moderate  technology,  all 
experienced  varying  degrees  of  difficulty  in 
meeting  cost  estimates  and  contractual  commit- 
men  t  s . 

A.  ORGANIZATIONAL  SLACK 

1.  Lack  of  incremental  (i.e.  milestone) 
development 

2.  Lack  of  control  of  the  entire  project 

3.  Overlapping  development  of  interde¬ 
pendent  projects 
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FIGURE  7  THE  CONTRACTING  PROCESS 
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4.  Split  and  dispersed  organizational  con¬ 
trol  of  key  elements  in  a  high  risk 
project  (i.e.  fragmentation  of  resp¬ 
onsibilities) 

5.  Inadequate  consideration  given  to 
trouble  areas  (i.e.  subsystem  depend¬ 
encies)  which  could  delay  the  program. 

6.  Incomplete  preplanning 

7.  Lack  of  organizational  cohesiveness 
and  continuity 

8.  Lack  of  adequate  manpower  to  deal  with 
design  changes 

9.  Incongruant  personnel  career  object¬ 
ives  . 

B.  CUSTOMER  URGENCY 

10.  Inadequacy  (or  incompleteness)  of  task 
definition  at  the  time  of  contract 

11.  Contracting  simultaneously  for  cost, 
time,  and  technical  performance 

12.  Contract  provisions  (e.g.  elimination 
of  contingency  provisions),  (e.g.  total 
package  concept),  and  negotiation 
techniques 

13.  Mutual  acceptance  of  unrealistic  pros¬ 
pective  cost  estimates  of  product  and 
delivery  schedules  (i.e.  cost  optimism 
sydrome,  schedule,  and  risk  optimism) 

14.  Over  lap  of  development  and  production 
phases 


tion  of  programs  such  as  those  described  in 
Table  6.  It  would  be  unreasonable  to  prevent 
advances  in  state-of-the-art  and  changes  in  de¬ 
sign  to  meet  requirements  that  are  determined 
during  system  test  and  evaluation.  Further¬ 
more,  there  will  inevitably  be  some  degree  of 
concurrency  during  development,  as  well  as 
overlap  of  authority  because  of  multiple  organi¬ 
zation  involved  in  the  process .  There  is 
little  doubt  that  over-optimism  in  new  designs 
lead  to  design  changes  and  ultimately  to  cost 
overruns.  Inflation,  changing  political  and 
customer  influences  and  environmental  castas - 
trophies  will  continue  to  plague  the  acquisi¬ 
tion  process.  Low  bidding,  poor  or  inappro¬ 
priate  estimates,  improper  budgeting  and  cost 
control — all  contribute  to  the  problem.  When 
these  factors  are  compounded  with  interrelated¬ 
ness,  delays,  disruption,  concurrency  and  ex¬ 
treme  variability,  it  is  little  wonder  that  no 
definitive  answer  has  emerged  to  solve  the  cost 
overrun  problem.  What  is  needed  is  a  positive 
approach  to  the  ,,management,,  of  risk  and  un¬ 
certainty  so  that  the  polemics  of  acquisition 
disruption,  as  described  by  Cochran  in  Table  7 
are  avoided.  (Shown  next  page) 


development  of  an  approach  to  the  management  of 

ACQUISITION  UNCERTAINTY 


C.  ENVIRONMENTAL  UNCERTAINTY 

15.  Lack  of  financial  strength  for  large, 
long-term  risky  projects 

16.  Low  bidding,  while  lacking  the  re¬ 
sources  to  finish  the  job 

17.  Political/economic  pressure  to  win  the 
competition  at  any  cost 

18.  Economic  pressures  for  a  general  re¬ 
duction  of  expenditures 

19.  Inflation,  regulation,  and  poor  cost 
estimates 

20.  Optimistic  promises  concerning  sched¬ 
ule,  cost,  and  technical  performance 

D.  TECHNOLOGICAL  UNCERTAINTY 

21.  Underestimation  of  the  degree  of  tech¬ 
nological  breakthrough  required  in  a 
state-of-the-art  of  product  develop¬ 
ment,  while  under  a  fixed  and  tight 
time  and  performance  constraints 

22.  Pushing  technology  too  fast 

23.  Lack  of  prototype  development 

24.  Performance  requirements  beyond  state- 
of-the-art 

25.  Inadequate  test  program 

26.  Major  design  or  scope  changes 

TABLE  4  CAUSE  OF  COST  OVERRUNS 
(See  .Page  34) 

Examination  of  the  data  from  the  many  studies 
conducted  reveals  that  there  are  no  simple 
answers  to  reducing  cost  growth  when  dealing 
with  the  uncertainty  inherent  in  the  acquisi¬ 


In  the  sections  that  follow,  we  will  attempt  to 
define  terminology  used  in  the  management  of 
risk  and  uncertainty  in  the  acquisition  of  maj¬ 
or  programs.  Patterns  of  disruption  will  be 
examined  in  terms  of  predictive  causality  mod¬ 
els.  Finally,  these  will  be  combined  with  a 
proposed  simulation  model  that  incorporates  the 
causal  models  and  that  treats  the  factors  ex¬ 
plicitly  contributing  to  cost  overruns  so  that 
alternative  policies  and  approached  can  be  ex¬ 
amined  in  terms  of  minimizing  expected  disrup¬ 
tion. 

Risk  and  uncertainty  are  defined  based  on 
classical  probability  theory  as  well  as  how 
they  apply  to  the  acquisition  process.  Lev 
(33)  defines  risk  as  the  condition  where  each 
outcome  of  the  decision  maker  leads  to  one  of  a 
set  of  possible  specific  outcomes,  each  occur¬ 
ring  with  a  known  probability.  Uncertainty  is 
defined  as  the  situation  where  the  probabili¬ 
ties  of  the  various  outcomes  are  completely  un¬ 
known.  Although  risk  and  uncertainty  are  often 
used  interchangeably,  they  are  not  the  same 
state  of  knowledge  in  a  given  situation. 

Peck  and  Scherer's  (145)  comprehensive  analysis 
of  the  weapon  systems  acquisition  process  de¬ 
fined  risk  as  the  level  of  consequences  of  a 
wrong  prediction.  They  operationally  defined 
uncertainty  as  the  relative  unpredictability  of 
an  outcome  of  a  contemplated  action.  They  cat¬ 
egorize  uncertainty  as  either  internal  or  exter¬ 
nal  where  internal  uncertainty  related  to  the 
possible  incidence  of  unforeseen  technical 
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Table  6 

Cgmsal  Factors  Contributing  to  Cost  Overruns 


Project 

Organizational 

Slack 

Technological 

Uicertainty 

Customer 

Urgency 

Environmental 

Uncertainty 

RB  ni- 
Rolls  Rocyce 
Engine 

Engine  thrust  be¬ 
yond  prior  exper¬ 
ience,  accepted  tn- 
realistic  deadline, 
lacked  financial 
strength 

20t  fewer  rotating 
parts,  first  3  spool 
shaft,  25  hyfil  blades, 
fan  failure  due  to  new 
titanian  alloy,  higher 
engine  weight  due  to 
severe  loading,  high 
temperatures  prevented 
early  testing. 

late  redesign 
to  reduce 
weight,  major 
design  changes, 
urgency  in 
delivery  dead¬ 
line 

bankruptcy  due 
to  unrealistic 
estimates 

LHA  - 

Landing  Heli¬ 
copter  Assault 
ship 

Management  and 

engineers  2000  ai. 
from  ship  yard, 
high  management 
turnover,  over  8 
year  period  had  7 
program  managers,  7 
yard  managers,  5 
chief  engineers 

Extremely  large  boilers 
rear  loading  ramp,  2 
conventional  ships  aid 
remainder  modularized 

large  number  of 
change  orders, 
shipyard  com¬ 
pleted  2  years 
after  contract 
award ,  CO  ship 
and  LHA  used 
same  yard 

Social  and  car- 
muni  ty  conflict, 
difficulty  to 
transfer  management 
to  the  yard 

BART  - 

Rapid  Transit 
System 

new  organization 
to  manage  construc¬ 
tion  and  then 
operate  the  system 

new  self  propelled 
railcar,  new  sophisti¬ 
cated  automatic  train 
control 

new  govemaent 
organization 
involving  the 
Association  of 
Several  Bay 

Area  entitities. 
Rohr  and  West- 
ing house  both 
had  significant 
concurrency 

system  reouired  both 
subway  and  elevated 
facilities  in  com¬ 
plex  environment 

nil  - 

Swing  wing 
fighter  air¬ 
craft 

Two  different  can- 

panics  designed  and 
produced  the  air¬ 
frame,  separate 
engine  development, 
Secretary  of 

Defense  took  per¬ 
sonal  charge  of  the 
project 

Major  advance  in  the 
state  of  the  art,  swing 
wing,  fully  modulated 
fan  jet,  self  contain¬ 
ed  crew  compartment, 
terrain  foilcaving  cap¬ 
ability,  1226  changes 
in  specifications,  pocrr 
product  definition 

Overlapping 
development  of 
wing  and  engine 
development, 
integration  of 
pheonix  and 

SRAM,  concurr¬ 
ency  of  produc¬ 
tion  and  deve¬ 
lopment 

Delavs  in  bidding 
changes  in  quantity 
ordered 

SRAM 

Short  range 

attack 

missile 

Coordination  be¬ 
tween  SRAM  office, 
Fill  development 
and  SPO 

New  alloys  and  new  de¬ 
sign  concept,  ambitious 
SQA  solid  rocket  pro¬ 
pellant  motors,  chamber 
pressure  and  bum  rate 

8  tines  that  of  the 
Minute-man  1 04  f  low 
radar  cross  section 

Use  of  the  Total 
Package  Produre- 
ment,  concurrent 
development 
with  FB111  air¬ 
craft,  lack  of 
cqgaonality  in 
launch  platforms 

Delays  in  coord¬ 
inating  tests, 
Switch  from 

TPP  concept 

CSA 

cargo  air  - 
craft 

Significant 
problems  of 
coordination  with 
the  SPO 

6200  square  feet  of  ad¬ 
ditional  wing  span,  in¬ 
tegration  of  subsystems, 
by  far  the  largest 
engines  ever  built, 
kneeling  landing  gear, 

28  tires,  automatic 
weight  balance 

Major  program 
changes ,  quantity 
changes,  over¬ 
lap  of  develop¬ 
ment  and  pro¬ 
duction  could 
not  test  engine 
because  of  over¬ 
lap  with  wing 
design 

Viet  Nar  War 
causing  50%  of 
overrun  due  to 
inflation 
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TABLE  7  THE  PUtiMJCS  OF  ACQUISITION  DISRUPTION 


Basic  Event 

The  Contractor's  Position 

The  Customer's  Position 

DESIGN  DELAYS 

Customer  Indecision 

Customer  Changes 

Regulatory  body  approval  cycla 

Insufficient  Contractor  rssourcea  assigned 

Other  deaign  work  given  priority 

Poor  engineering  work  required  Customer  correction 

Poor  Contractor  communication  with  Customer 

Contractor  didn't  follow  instructions 

DESIGN  CHANCES 

Cuatomer  changed  project’s  objec¬ 
tives  or  scops  to  meet  new 
competition 

Customer  misrepresented  the  task 
Required  to  correct  commercial 
Infeasibility 

Contractor  knew  such  changes  would  occur  and  mads  pro¬ 
vision  in  his  price  and  planning 

Changes  made  to  help  Contractor  meet  schedule 

Poor  Contractor  design  work  had  to  ba  correctsd 

Contractor  failed  to  follow  Customer  specs. 

Host  design  work  actually  dons  by  Customer 

POOR  PERFORMANCE 

Meets  Customer  design  specs. 

Degradation  due  to  customer 
dasign  changes 

Not  enough  time  allowed  by  cua¬ 
tomer  for  testing 

Cuatomer  didn't  train  its  field 
personnel 

Poor  deaign  work  by  Contractor 

Psils  to  meet  Cuatomer  performance  specs. 

Poor  Contractor  quality  control 

Contractor  failed  to  determine  design  changs  impact 
Contractor  manuals  lnadequata 

LATE  DELIVERY 

Customer-caused  deaign  delays/ 
changes  affected  vendors, 
design  or  production 

Force  majeure  (no  one's  fault) 

See  reasons  for  design  delays /chsnges 

Poor  Contractor  planning 

Poor  aalectlon  of  or  coordination  with  vendors 

Production  inefficiency 

Poor  quality  control 

Force  majeure  (Contractor  responsibility) 

COST  OVERRUN 

Design  delays 

Deaign  changes 

Cuatomer changed  delivery  achedula 
Interactive  disruption 

Escalation  dua  to  Cuatpmar  dalays 
Force  majeure 

Grosa  Contractor  undsr-sst lmatlng 

Contractor  "bought"  the  contract 

Operating  lna f f lclsncles ;  weak  management  control 
Escalation  due  to  Contractor  delays 

Poor  planning 

Cenaral  mismanagement 

Mlststement  of  costa  by  Contractor 

DELIVERY  ACCELERATION 

Cuotomer  convenience 

Contractor  should  have  sxpseted  this 

DELIVERY  STRETCHOUT 
(OR  CANCELLATION) 

Customer  convenience:  misjudged 
market'  shortage  of  funds 

Schedule  wsa  adapted  to  Contractor  capabilities 

Contractor  delsya/qusllty  esuaed  lower  sales 

DISRUPTION  OF  OTHER 
PRODUCTS 

Reflects  problems  caused  by  cus¬ 
tomer  on  this  product  (See  above) 

Contractor  attempt  to  cover  own  errors 

f 


difficulties  in  the  development  of  a  specific 
weapons  system.  Examples  of  internal  uncer¬ 
tainty  include  development  time  of  interrelated 
technologies,  substitutable  technologies,  and 
performance  to  specification.  External  uncer¬ 
tainty  covers  factors  external  to  a  given  pro¬ 
ject,  but  affect  the  course  and  outcome  that 
can  be  expected.  Examples  include  rate  of 
technological  change  in  weaponry,  changes  in 
strategic  requirements  and  shifts  in  govern¬ 
ment  policy. 

The  USAF  Risk  Analysis  report  defines  risk  as 
the  probability  of  an  occurrence  and  uncertain¬ 
ty  as  incomplete  knowledge.  A  risk  assessment 
is  where  estimates  are  made  of  the  risk  asso¬ 
ciated  with  given  alternatives  and  risk  manage¬ 
ment  as  the  actions  taken  to  reduce  risk.  Risk 
analysis  is  considered  the  combination  of  risk 
assessment  and  risk  management.  It  is  the 
latter  definitions  which  are  most  directly  app¬ 
licable  to  the  acquisition  process.  As  was 
shown  previously,  they  use  uncertainty  to  des¬ 
cribe  target,  technical,  internal  program  and 
process  effects.  They  also  use  a  network  simu¬ 
lation  to  develop  individual  and  joint  risk 
profiles  as  the  system  progresses  over  time. 

Harrison  (20)  defines  risk,  certainty,  and  un¬ 
certainty  as  follows: 

risk  -  a  common  state  or  condition  in  deci¬ 
sion-making  characterized  by  the 
possession  of  incomplete  informa¬ 
tion  related  to  a  probabilistic  out¬ 
come  . 

certainty  -  an  uncommon  state  of  nature 

characterized  by  the  possession 
of  perfect  information  related 
to  a  known  outcome. 

uncertainty  -  an  uncommon  state  of  nature 
characterized  by  the  absence 
of  any  information  related 
to  a  desired  outcome. 

Harrison  further  contends  that  "genuine  uncert¬ 
ainty  is  as  common  as  complete  certainty".  The 
more  common  state  of  nature  is  incomplete  or 
imperfect  information,  which  means  that  the  ex¬ 
pected  outcome  contains  an  element  of  risk  for 
the  decision  maker.  There  is  no  situation  that 
deals  with  the  future  that  can  be  completely 
known  when  the  acquisition  process  lasts  any¬ 
where  from  2  to  12  years.  How  can  a  program 
manager  possibly  forecast  events  that  far  in 
the  future  with  any  meaningful  degree  of 
accuracy? 

Beverly,  et  al,  (86)  describe  uncertainty  in 
systems  acquisition  as  the  lack  of  knowledge 
in  development  requiring  state-of-the-art  tech¬ 
nology.  They  apply  risk  based  on  historical 
phenomena  for  which  probabilities  can  be  est¬ 
ablished.  On  the  other  hand,  certainty  or  un¬ 
certainty  deals  with  the  existence  of  know¬ 


ledge.  Uncertainty  is  greatest  when  knowledge 
is  at  its  lowest  level.  Uncertainty  would 
describe  the  situation  where  a  new  system  is 
being  developed  which  involved  advanced  state- 
of-the-art  technology.  The  lack  of  knowledge, 
in  turn,  inevitably  leads  to  errors  in  estima¬ 
ting,  in  design  and  ultimately  in  cost  control. 
This  leads  to  three  kinds  of  uncertainty  in 
weapons  acquisition:  design  and  technology  un¬ 
certainty,  scheduling  uncertainty  and  cost 
uncertainty.  They  point  out  that  there  is  con¬ 
flict  among  goals  because  reduced  design/tech- 
nology  uncertainty  enhances  performance  while 
cost  minimization  tends  to  adversely  affect 
both  performance  and  schedule  goals. 

Martin  (134)  deals  with  uncertainty  in  terms  of 
our  inability  to  predict  the  future  in  the  face 
of  unknown  variables.  His  taxonomy  of  uncert¬ 
ainty  conditions  represents  a  comprehensive 
treatment  of  the  subject.  He  includes  four 
basic  categories  of  uncertainty  as  follows: 

1.  Environmental: 

a)  natural  factors 

b)  social  &  political  effects 

c)  communication  disparities 

d)  time  which  results  in  distortions 

e)  external  to  the  project  or  exogenous 

f)  internal  approaches  or  endogenous 

2.  Functional: 

a)  income/business  risk 

b)  financial/earnings  risk 

c)  technological  uncertainty 

d)  production  inadequacies 

3.  Informational: 

a)  unknows  of  which  contractor  is  aware 

b)  unknowns  that  cannot  be  foreseen 

c)  lack  of  knowledge 

d)  unknowns  that  cannot  be  anticipated 

4.  Technical: 

a)  uncertainty  -  no  known  probability  dis¬ 
tribution  of  events 

b)  risk  -  outcomes  can  be  described  by  a 
probability  distribution 

c)  certainty  -  predictable  outcome  deter¬ 
mined 

d)  subjective  -  probabilities  derived  in¬ 
dependent  of  the  problem  at  hand 

Martin  (134)  describes  a  twenty  year  period  in 
which  measures  to  reduce  cost  growth  were  not 
effective.  He  recommends  the  use  of  entropy  to 
measure  the  level  of  information  in  a  system 
which  is  directly  related  to  the  uncertainty 
under  which  decisions  have  to  be  made.  As  en¬ 
tropy  increases,  so  does  uncertainty  and  what 
is  needed  is  a  means  to  increase  information 
efficacy  rather  than  increase  choices  or  ran¬ 
domness.  Thus,  if  order  is  complete  (reduced 
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entropy)  cost  is  known:  whereas,  if  multiple 
outcomes  are  possible,  cost  is  uncertain  and 
difficult  to  control. 

McNichols  (138)  presents  a  means  for  estimat¬ 
ing  the  distribution  of  cost  uncertainty  where 
actual  costs  differ  significantly  from  orig¬ 
inal  cost  estimates.  He  contends  that  cost 
overrun  is  a  meaningless  concept  because  all 
cost  estimates  relay  heavily  on  subjective 
judgments  and  are  subject  to  considerable  un¬ 
certainty.  He  considers  four  basic  require¬ 
ments  in  the  treatment  of  uncertainty.  These 
include:  generation  of  probability  distribu¬ 

tion  for  individual  cost  elements;  generate  a 
total  cost  by  additive  distributions;  the  pro¬ 
bability  density  functions  are  then  combined 
to  form  a  compound  distribution;  finally  the 
dependence  or  degree  of  correlation  between 
cost  elements  is  taken  into  account.  The  pro¬ 
blem  of  uncertainty  then  is  to  determine  a 
measure  of  the  degree  of  difficulty  or  likeli¬ 
hood  of  achieving  cost  goals. 

The  descriptions  of  risk  and  uncertainty  pre¬ 
sented  above  illustrate  the  variety  of  ap¬ 
proaches  that  can  be  taken.  The  relevant 
question,  however,  is  how  best  can  management 
deal  with  the  problem  of  uncertainty  in  the 
acquisition  process.  Although  it  is  commonly 
assumed  that  any  major  overrun  signifies  poor 
management,  this  premise  fails  to  recognize 
that  uncertainty  is  inherent  in  acquisition 
and  that  managers  operate  under  severe  time 
and  resource  constraints. 

Even  where  the  degree  of  uncertainty  can  be 
quantified,  either  by  using  past  experience  or 
by  applying  analytical  techniques  on  the  basis 
of  a  supportable  theory,  there  is  a  significant 
probability  of  failure.  In  accepting  risk,  the 
trade-off  must  be  clearly  assessed.  For  ex¬ 
ample,  economic  growth  is  worth  less  to  some 
manager  (or  their  capital  sources)  than  to 
others.  Thus,  arriving  at  a  rational  value  of 
specific  losses  and  benefits.  Thus,  there  is 
a  need  for  a  "utility  function"  which  identi¬ 
fies  the  financial  consequences  of  a  decision. 

In  view  of  the  inherent  uncertainty  in  the  ac¬ 
quisition  process,  managers  must  consider:  a) 
did  the  project  warrant  the  resources  and  risks 
involved;  b)  were  the  alternatives  rationally 
defined;  c)  were  considerations  given  to  the 
probable  results  of  alternatives  as  to  the 
failure  of  the  project;  d)  were  suitable  steps 
to  prevent  failure  or  to  hold  cost  to  a  mini¬ 
mum  made  available  and  e)  was  there  adequate 
monitoring  of  events  to  detect  deterioration  of 
the  situation  in  an  early  stage,  and  to  limit 
the  losses. 

Clearly,  managers  take  risks  because  they  pro¬ 
ceed  in  the  face  of  possible  failure.  Unfort¬ 
unately,  the  "degree"  of  uncertainty  and  the 
potential  effects  of  unfavorable  events  are 
often  difficult  to  determine.  Many  situations 


involve  potential  events  whose  probability  is 
not  measureable.  All  too  often  the  term  "cal¬ 
culated  risk"  refers  to  a  decision  reached  in 
recognition  of  factors  known  to  be  unfavorable 
but  "not"  susceptible  to  calculation.  The 
future  may  even  involve  events  of  a  totally  un¬ 
expected  or  unknown  type.  Thus,  in  dealing 
with  complex  problems  under  limited  time  and 
resources,  substantive  errors  in  judgment  can 
occur  in  unpredictable  ways. 

A  CAUSAL  BASIS  FOR  DEFINING  UNCERTAINTY 

Although  uncertainty  is  defined  as  lack  of 
knowledge  about  specific  effects,  it  can  be  ex¬ 
amined  in  terms  of  the  factors  that  contribute 
to  disruption  and  in  turn  attempt  to  understand 
the  causal  relations  that  lead  to  cost  increase. 
The  premise  is  that  control  of  the  variables 
contributing  to  uncertainty  is  an  effective 
means  for  controlling  cost.  This  is  analogous 
to  queuing  theory  where  a  knowledge  of  queue 
behavior  and  sequencing  rules  permits  ser¬ 
vicing  the  maximum  demand  with  available  re¬ 
sources.  Delays  are  not  eliminated;  rather, 
they  are  reduced  by  adding  capacity  or  modified 
by  changing  priority  rules.  Disruption  in  the 
acquisition  process  can  be  considered  similar 
to  queuing  delays  in  limited  capacity  servers. 
By  understanding  which  factors  cause  disrup¬ 
tion,  management  can  alter  the  expected  cost 
growth  by  controlling  those  factors.  Typical 
factors  leading  to  disruption  are  shown  in 
Table  8. 

1.  Delay:  gaps  in  carrying  out  a  program 

2.  Interruption:  short  delay 

3.  Stretch-out:  slow  down  of  program 

4.  Interference:  delay  by  stoppage 

5.  Redesign:  change  scope,  redo  previous 

work 

6.  Work  stoppage:  partial  interruption 

7.  Interdependencies:  indirect  delays 

8.  Shortages  or  errors:  delay  due  to  re- 

wo  rk 

9.  Overlay:  interferences  &  delay  due  to 

concurrency 

10.  Redirection  of  effort:  disruptive 
effect  of  reorganization 

TABLE  8  FACTORS  IN  DISRUPTION 

Two  key  factors  that  are  highly  disruptive  in 
the  acquisition  process  involve  concurrency  and 
technological  uncertainty.  Concurrency  is  most 
often  a  result  of  customer  urgency  in  attempt¬ 
ing  to  meet  tight  deadlines.  Delivery  urgency 
enforced  by  competitive  conditions  exerts 
strong  pressure  on  suppliers  to  commit  to  deli¬ 
very  dates  which  are  inherently  optimistic  or 
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based  on  the  assumption  that  no  serious  pro¬ 
blems  will  develop.  The  plan  becomes  a  matter 
of  urgency  when  combined  with  some  degree  of 
technological  uncertainty. 

Delivery  urgency  can  be  disruptive  even  when 
the  initial  delivery  cannot  be  eased,  since 
planning  is  done  and  work  initiated  predicated 
on  the  original  delivery  date.  A  deferral  in 
scheduled  delivery  causes  added  costs. 

TECHNOLOGICAL  UNCERTAINTY 

As  used  here,  technological  uncertainty  refers 
to  two  conditions.  One  is  the  highly  abstruse 
demands  at  the  very  forefront  of  scientific 
knowledge  or  state-of-the-art.  It  also  refers 
to^a-major  gap  between  an  organization’s  area 
of  expertise  and  what  is  required  to  perform 
effectively.  Rapid  technological  change  can 
have  a  major  financial  impact  on  an  organiza¬ 
tion  which  can  be  catastropic  and  can  be  term¬ 
ed  a  ’’technical  disruption." 

Uncertainty  occurs  where  conditions  of  rapid 
technological  change  exist  simply  because  it 
is  difficult  to  make  valid  judgments.  Even  ex¬ 
perts  have  incomplete  knowledge.  Managers,  in 
turn,  must  rely  on  the  recommendations  of 
technical  people,  and  yet  be  able  to  detect 
errors  and  inconsistencies.  Hence  the  mana¬ 
ger’s  incomplete  knowledge  can  be  a  limiting 
factor . 

During  a  period  of  rapid  technological  change, 
there  is  usually  intense  competition,  since  a 
major  advance  by  one  enterprise  reduces  the 
business  available  to  others.  As  a  result, 
the  amount  of  time  available  for  testing  and 
proving  of  new  concepts  and  hardware  is  held 
to  a  minimum.  Factors  used  to  determine  the 
state-of-the-art  are  shown  in  Table  9. 

1.  Size-number  of  interrelated  components, 
physical  volume 

2.  Complexity-difficulty  in  meeting  per¬ 
formance  requirement 

3.  Newness  of  technology-experimental  state 
of  technology 

4.  Percent  proven  technology-degree  of 
newness 

5.  Experience  in  the  field-work  on  similar 
programs 

6.  Percent  new  components-tes t  and  evalua¬ 
tion  requirements 

7.  Interdependency  of  subsystems- types  of 
linkages 

8.  Degree  of  precision-quality  or  cleanli- 
nt’KM  requirements 

9.  Special  resources-testing,  or  tooling 
requirements 

10.  Definitive  specif ications-clarity  in 
meeting  requirements 

11.  Design  flexibility- tolerance  level,  sub¬ 
stitutes  available 

12.  Required  theoretical  analysis-need  to 
support  proposed  design 


13.  Degree  difference  from  existing  tech¬ 
nology-life  cycle  of  technology 

14.  Available  knowledge  in  the  field- 
amount  of  experimentation  required 

15.  Infra-structure  support  required- 
degree  of  dependency  on  vendors 

TABLE  9  FACTORS  DETERMINING  THE 
STATE-OF-THE-ART 

An  approach  to  determining  the  technological 
advance  ratings  was  developed  at  the  RAND  Corp¬ 
oration  (146)  and  is  shown  in  Figure  8.  The 
scale  ranges  from  0  to  20  where  the  newness  of 
the  design  determines  the  advance  in  state-of- 
the-art.  Examples  of  a  number  of  military  and 
commercial  aircraft,  as  well  as  a  number  of 
different  missiles  is  shown  on  the  chart.  The 
factors  shown  in  Table  9  include  the  newness 
as  well  as  the  design  requirements  for  deter¬ 
mining  the  state-of-the-art.  Thus,  state-of- 
the-art  for  a  given  organization  can  be  con¬ 
strued  as  the  "ability"  to  produce  a  given 
design,  in  addition  to  the  newness  of  the 
technology  involved.  (Shown  page  41) 

Uncertainty  due  to  technological  change  often 
arises  from  the  overlap  or  "concurrence”  of 
development  and  production.  The  perceived 
necessity  to  initiate  the  ponderous  and  in¬ 
volved  processes  of  production  before  there  is 
real  certainty  as  to  the  stability  of  the  pro¬ 
duct  design,  places  these  at  the  mercy  of  any 
delays  or  changes  which  may  occur  in  the  de¬ 
sign.  And,  such  delays  or  changes  are  more 
likely  to  occur  as  the  degree  of  concurrency 
increases . 

Concurrency  is  inherently  costly  since  it  re¬ 
quires  considerable  effort  and  cost  to  estab¬ 
lish  production  momentum,  and  sometime  even 
more  to  shut  it  off  or  slow  it  down.  A  pro¬ 
duction  organization  is  not  very  good  at  res¬ 
ponding  to  the  inherently  sporadic  character 
of  a  development  program,  and  its  floundering 
is  dreadfully  expensive. 

It  might  seem  obvious  that  the  solution  is  tc 
avoid  concurrency,  or  at  least  to  limit  it  to 
areas  in  which  redundant  or  alternative  courses 
of  action  offer  an  option  with  which  to  avoid 
real  failure.  Although  this  may  be  true,  it 
is  no  simple  matter  to  implement. 

During  rapid  technological  change,  the  pres¬ 
sure  to  take  on  new  projects  is  greatest.  But, 
doing  so  entails  even  greater  risk  than  under 
conditions  of  technological  stability.  Un¬ 
fortunately,  there  is  no  warning  that  "this" 
risk  will  turn  out  to  be  unacceptable.  The 
effects  are  pervasive  and  far  reaching,  rang¬ 
ing  from  the  ineffectiveness  of  familiar  tech¬ 
nical  skills  and  operational  procedures  to 
the  impact  of  recondite  scientific  laws  on  the 
design*  construction,  operation  and  mainten¬ 
ance  of  a  new  product. 

In  an  all  too  common  form,  technological 
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FIGURE  8  TECHNOLOGICAL  AOVANCE  RATINGS 
(SOURCE:  RAND  R-733-PR/ARPA  JUNE  1971) 
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disruption  has  the  following  characteristics: 

— The  enterprise  embarks  on  a  program  re¬ 
quiring  the  delivery  of  a  specific  pro¬ 
duct  within  a  specified  time  period. 

— The  product  to  be  delivered  is  not  fully 
designed,  but  is  thought  to  involve  only 
normal  state-of-the-art,  familiar  to  the 
organization. 

— Unanticipated  technological  problems  arise 
which  require  extensive  time  to  resolve 
and  result  in  substantial  changes  to  the 
original  product  and  to  production. 

— The  changes  generate  confusion  and  in¬ 
crease  costs  substantially,  and  those  in¬ 
creases  are  intensified  by  the  need  for 
skills  and  training  and  for  additional 
management  time. 

—  Intensive  efforts  must  be  made  to  minimize 
deviation  from  the  original  delivery 
schedule  and  product  specifications.  This 
further  increases  costs,  as  the  natural 
confusion  caused  by  change  itself  is  com¬ 
pounded  by  the  need  to  expend  greater 
effort  in  the  shrunken  time  now  available 
for  design  and  production,  and  by  the  need 
to  invent  new  procedures  to  make  better 
use  of  the  time. 

Much  interaction  occurs  among  the  various 
stages  and  events  of  each  situation,  and  the 
end  result  can  be  a  cost  overrun  of  enormous 
proportions . 

INTERDEPENDENCY 

If  the  degree  of  state-of-the-art  is  a  driver 
of  technological  uncertainty,  then  interre¬ 
latedness  is  a  major  multiplier  on  cost  of 
development  and  production.  Interrelatedness 
of  design  boils  down  to  the  fact  that  a  change 
in  one  component  or  subsystem  affects  many 
others,  and  of  course  this  process  can  iterate 
further.  Interrelatedness  can  also  affect 
production  and  vendor  activities,  since  a 
change  in  production  methods  or  delivery  cycle 
in  one  area  or  component  (generated,  or  course, 
by  design  delays  and  changes)  may  affect  pro¬ 
duction  of  other  components  or  work  in  other 
areas.  Again  the  process  can  become  very 
complex  with  many  ripple  effects.  In  general, 
a  product  operating  in  a  more  advanced  area 
of  technology  will  be  more  subject  to  inter¬ 
relatedness.  Interrelatedness  can  be  des¬ 
cribed  in  a  matrix  form  as  shown  in  Figure  9. 
(Shown  previous  page) 

ORGANIZATIONAL  SLACK 

Two  other  factors  compound  the  disruptive  ef¬ 
fect  of  concurrency:  First;  the  level  of  re¬ 
sources  effectively  available  to  the  project 
and  second,  the  degree  of  external  control 
over  events.  The  level  of  resources  comprises 
all  types  of  resources  —  technical,  manager¬ 
ial,  facilities,  financial,  etc.  Adequacy  of 
resources  is  measured  by  what  might  be  termed 
“organizational  slack“. 


This  factors  relates  to  the  organization’s  ex¬ 
perience  in  the  basic  technology  involved.  It 
provides  an  invaluable  fund  of  knowledge  and 
skill  in  handling  the  inevitable,  unexpected 
problems  which  arise,  and  which  could  swamp  an 
inexperienced  organization.  A  second  problem 
is  the  degree  to  which  the  task  at  hand  fully 
engages  all  resources  available  over  the  time- 
frame  of  the  project,  which  may  leave  inade¬ 
quate  reserves  for  use  on  unexpected  problems. 
This  inadequacy  can  be  a  critical  flaw,  given 
the  intense  time  compression  inherent  in  con¬ 
current  design  and  production. 

Organizational  slack,  thus,  defines  the  level 
or  degree  of  unknowns  that  are  internal  to  the 
system  rather  than  the  external  exigencies. 
Factors  related  to  internal  uncertainty  could 
be  measured  using  dimensions  such  as: 

1.  The  organizations  ability  to  respond  to 
new  or  unforeseen  requirements. 

2.  The  slack  or  flexibility  that  has  been 
built  into  the  organization. 

3.  Prior  experience  with  the  given  techno¬ 
logy. 

4.  Number  of  linkages  of  subsystem  depend¬ 
encies  or  interaction  with  other  projects. 

5.  Percent  of  the  project's  subsystems  being 
developed  that  are  at  the  "state-of-the- 
art"  of  the  technology. 

6.  The  amount  of  time  compression  or  tight¬ 
ness  of  schedules  (concurrency) . 

7.  Availability  of  or  access  to  resources. 

8.  Maturity  in  the  planning  and  control  of 
operations,  including  computer  systems 
and  organization  structure. 

9.  Amount  of  overlap  of:  development, 
design,  and  implementation. 

10.  Number  of  contractors  or  organizations 
involved  in  the  project. 

These  factors  contribute  to  a  measure  called, 
"Organization  Response  Capacity."  That  is, 
management's  ability  to  cope  with  uncertainty. 
In  turn,  the  delay,  disruption,  or  slippage 
that  can  be  anticipated  would  be  measured  by 
the  relationship  of  this  capacity  to  custo¬ 
mer  demand  as  shown  in  Figure  10. 

Expected  disruption  is  an  exponentially  in¬ 
creasing  function  which  is  dependent  on  the 
organizational  response  capacity,  which  in 
turn  depends  on  the  level  of  concurrency. 

Thus,  when  the  level  of  concurrency  approaches 
the  response  capacity,  the  delay  increases. 
This  formulation  does  not  deal  with  uncertain¬ 
ty  per-se,  but  whether  the  organization  is 
able  to  cope  with  problems  as  they  arise,  or 
is  able  to  anticipate  problems.  In  turn,  the 
amount  of  slack  or  flexibility  in  the  organi¬ 
zation  determines  the  ability  to  respond  to 
uncertain  requirements.  If  management  is 
operating  with  minimum  slack,  then  any  dis¬ 
ruption  causes  a  large  delay. 

(See  figure  10  and  Table  10  next  page) 
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TABLE  10  -  EXAMPLES  OF  PROGRAM  MANAGEMENT 
(SOURCE  RAND  -  R-1375,  APRIL  1975) 
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Another  perspective  of  management  practices  is 
shown  in  Table  10  for  four  government  agencies 
based  on  a  RAND  (54)  study  of  R&D  management. 

An  examination  of  the  findings  reveals  the 
considerable  latitude  given  program  managers 
in  dealing  with  creative  individuals  needed  in 
R&D  programs.  Given  this  kind  of  organization¬ 
al  environment,  the  accuracy  of  estimates  is 
highly  questionable.  At  best,  the  estimate 
becomes  a  target  that  permits  a  level  of  ef¬ 
fort  in  attempting  to  achieve  what  are  often 
elusive  objectives  or  requirements. 

Perry,  et.  al,  in  their  study  of  acquisition 
strategies,  (53)  recommend  that  acquisition 
management  use  an  incremental  approach.  This 
support  was  based  on  an  analysis  of  36  major 
DoD  programs  which  revealed  that  high  cost 
growth  was  due  to: 

1.  Willingness  to  pay  the  price  for  having 
high  technology  with  compressed  sched¬ 
ules  . 

3.  Over-optimism  regarding  the  cost  of  cop¬ 
ing  with  long  term  technology. 

3.  Little  evidence  that  the  programs  had 
extreme  urgency. 

4.  Little  improvement  in  cost  based  on: 

a)  contractual  approaches 

b)  complex  management  reforms 

c)  improved  estimating 

d)  early  identification  and  correction 
of  cost  growth 

Despite  these  four  factors,  a  number  of  pro¬ 
grams  had  surprisingly  good  outcomes  and  were 
able  to  predict  cost  performance  and  schedule. 

Using  their  findings,  the  authors  suggest  that 
an  incremental  strategy  and  control  in  the 
early  phases  of  development  would  have  the 
most  effect  on  avoiding  cost  growth. 

The  incremental  strategy  recommend  the  follow¬ 
ing  steps: 

1.  Resolve  uncertainty  early  in  the  program 

2.  Avoid  concurrency  of  development  and  pro¬ 
duction 

3.  Separate  performance  from  reliability 
and  maintainability 

4.  Require  periodic  reassessment,  redefini¬ 
tion  and  readjustment  regarding  pro¬ 
posed  changes 

5.  Conduct  tradeoff  studies  to  resolve  re¬ 
structuring 

Tlio  belief!  t.s  frnm  an  incremental  approach  to 
management  would  lie  in  greater  predictability 
based  on  prototype  demonstration  and  in  un¬ 
covering  difficulties  early  in  program  life. 

It  would  also  encourage  competition  and  trans¬ 
fer  of  technology  as  the  need  required. 

Another  consideration  relating  to  organization¬ 
al  slack  is  presented  in  the  study  by  Moeller 
(45)  of  the  DSARC  management  review  process  to 


determine  its  effect  on  the  length  of  major 
system  acquisitions.  He  found  that  although 
DSARC  demands  considerable  time  and  generates 
a  sizeable  workload  for  the  program  office, 
there  was  no  excessive  delay  in  11  of  the  13 
programs  examined.  His  conclusion,  therefore, 
was  that  regardless  of  how  cumbersome  the  re¬ 
view  process  might  be,  it  had  no  significant 
effect  on  the  length  of  the  acquisition  cycle 
because  the  review  was  concurrent  with  the 
production  activities.  Rather,  the  primary 
contributer  to  lengthening  the  development 
process  was  lack  of  adequate  funding  or  in¬ 
stability  which  caused  stretch-outs.  Another 
significant  factor  in  lengthening  the  cycle 
was  the  lack  of  agreement  on  configuration 
and  performance  parameters.  This  lead  to  in¬ 
decision  or  inconsistency  in  meeting  techni¬ 
cal  requirements.  There  were  a  number  of  de¬ 
lays  resulting  from  testing  requirements.  Two 
significant  recommendations  were  the  judicious 
use  of  concurrency,  such  as  for  logistics  and 
more  flexibility  in  the  approach  to  acquisi¬ 
tion. 

ORGANIZATION  PERFORMANCE 

Another  aspect  of  organizational  slack  relates 
to  expected  performance.  Cochran  (96)  has 
identified  key  factors  which  contribute  to 
disruption  and  which  management  can  review  in 
order  to  achieve  more  effective  control: 

1.  CONTRIBUTORS  TO  TASK  VARIABILITY 

a)  Inadequate  definition  of  product 
specification . 

b)  Underestimating  the  degree  to  which 
"state-of-the-art "  (SOA)  must  be 
advanced. 

c)  Poor  cost  engineering  or  organiza¬ 
tion  planning. 

c)  Not  allowing  for  the  degree  of  un¬ 
certainty  in  meeting  plans. 

d)  Not  anticipating  the  "backup"  acti¬ 
vities  required  in  case  the  main 
approach  fails. 

2.  DETERMINING  DISRUPTION 

a)  Review  the  source  of  rigidity  in 
the  delivery  date. 

b)  Analyze  the  SOA  tradeoff  surface. 

c)  Determine  areas  where  tasks  were 
not  anticipated. 

d)  Define  the  degree  of  SOA  advance¬ 
ment  required,  and  the  cost  in¬ 
volved  by  area. 

e)  Determine  the  risk  elements  invol¬ 
ved  and  their  effects 

f)  Define  specific  cost  increase  re¬ 
lationships. 

g)  Develop  modeling  techniques  to  con¬ 
duct  appropriate  analysis. 

Considering  that  industry  is  often  confronted 
with  untenable  contractual  procedures. 
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including  perpetual  specification  change  and 
rigid  contracting  requirements,  as  well  as,  un¬ 
anticipated  price  changes,  inflation,  changes 
in  the  number  of  systems,  and  the  impact  of  new 
technology,  it  is  small  wonder  that  acquisition 
managers  are  confronted  with  the  requirements 
for  effective  means  for  handling  uncertainty. 

Cochran  (96)  also  described  the  S-Curve  pat¬ 
terns  of  labor  hours  as  a  cause  of  disruption 
leading  to  substantial  cost  overruns  when  dev- 
lopment  of  a  major  new  design  is  concurrent 
with  production  and  under  severe  delivery  pres¬ 
sure.  Labor  cost  reflects  the  impact  of  de¬ 
sign  delays,  growth,  and  changes  in  the  pro¬ 
duction  function.  The  S-Curve  developed  by 
Cochran  is  shown  in  Figure  11  and  includes  the 
effect  of  interruption  which  leads  to  forget¬ 
ting  and  subsequent  restarting  of  the  program. 

The  disruption  caused  by  the  S-Curve  effect 
generally  continues  well  beyond  the  first  units 
produced,  because  of  the  way  in  which  produc¬ 
tion  operates.  The  procedures,  tools,  and 
methods  established  during  the  start-up  period 
inevitably  carry  forward  to  subsequent  periods. 
Costs  follow  accordingly  and  managers  generally 
acknowledge  that  it  is  harder  to  revise  en¬ 
grained  organization  practices  than  it  is  to 
start  from  scratch.  Further,  design  growth 
and  changes  cause  revisions  to  production 
methods  and  sequencing,  and  in  facilities  us¬ 
age.  If  a  change  is  introduced  after  pro¬ 
duction  has  been  established,  considerable  time 
is  required  to  fully  implement  the  program.  If 
design  changes  occur  after  the  affected  com¬ 
ponents  have  been  completed,  this  requires  re¬ 
work  and  reinstallation,  which  involves  extra 
cost.  The  cost  of  such  work  is  dependent  on 
the  degree  to  which  it  is  different  from  the 
position  or  sequence  normally  assigned  to  the 
original  task.  Such  work  also  creates  exten¬ 
sive  interference  with  other  on-going  tasks, 
which  can  involve  corresponding  greater  cost 
penalties . 

Another  cause  of  disruption  carryover  is  the 
'’queuing  effect".  For  example,  work  still  in 
process  must  be  held  up  because  of  design  de¬ 
lays,  design  changes,  or  the  need  to  perform 
a  sizable  amount  of  rework.  Inventory  control 
demands  frequent  rearranging  to  locate  items 
currently  required  from  the  shop,  and  other 
double  handling  affects  units  in  process.  In 
turn,  the  clogging  up  of  valuable  staging  areas 
and  even  workspace  may  cause  direct  inter¬ 
ference  with  follow-on  units. 

The  repeated  delays  imposed  on  the  production 
organization  in  the  early  stages  of  a  new  pro¬ 
duct  cause  deceleration  of  previous  activities 
and  rework  with  their  many  cost  penalties.  But 
beyond  that,  the  relentless  need  to  deliver  on 
time  causes  a  corresponding  acceleration  later, 
with  its  own  cost  penalties.  The  repeated  cy¬ 
cles  of  deceleration-acceleration  generates  a 
pulsation  which  sweeps  across  every  phase  of 


production,  gaining  momentum  and  leaving  con¬ 
fusion  and  wasted  effort  in  its  wake.  The 
effects  on  production  procedures,  facilities 
utilization  and  personnel  deployment  and  mor¬ 
ale  are  profound,  and  account  for  much  of  the 
cost  overruns  and  schedule  slippages  encounter¬ 
ed  production.  These  effects  are  summarized 
in  Table  11.  (Fig.  11  &  Table  11  shown  next 
page) 

DETERMINING  A  PATTERN  OF  DISRUPTION 

The  ability  to  define  causal  relations  among 
variables  in  disruption  and  uncertainty  is  a 
first  step  in  predicting  cost  overruns  and  in 
determining  which  actions  a  program  manager 
should  take  to  avoid  cost  growth.  For  example, 
Augustine  (4)  proposed  using  additional  plan¬ 
ning  funds  based  on  an  assessment  of  risk.  He 
contends  that  even  the  most  capable  program 
manager  is  not  able  to  forecast  all  the  pro¬ 
blems  that  will  be  encountered  in  a  develop¬ 
ment  program  spanning  anywhere  up  to  ten 
years.  However,  it  is  quite  possible  to  fore¬ 
cast  the  "probability"  that  additional  funds 
will  be  required.  He  recommended  the  use  of 
TRACE  (Total  Risk  Assessing  Cost  Estimate)  as 
the  basis  for  justifying  the  additional  fund¬ 
ing 

One  of  the  early  attempts  to  deal  with  uncer¬ 
tainty  was  proposed  by  Marshak,  Glennon  and 
Summers  (132).  They  indicate  that  where  "com¬ 
ponent"  interrelatedness  is  defined,  one  can 
predict  the  effects  that  are  likely  to  occur. 
Under  condition  of  uncertainty,  low  slack 
heightens  interrelatedness  and  substantially 
increases  the  risk  of  redesign.  Furthermore, 
the  risk  of  redesign  is  sensitive  to  the 
degree  that  design  reaches  beyond  past  state- 
of-the-art  and  where  there  are  requirements 
to  use  existing  components  which  can  strain 
the  designer  and  lead  to  suboptimization. 

Based  on  three  conditions  describing  component 
interrelatedness,  one  is  in  a  position  to  pre¬ 
dict  potential  disruption.  When  there  is  a 
high  degree  of  close  coupling  or  interre¬ 
latedness,  the  likelihood  of  design  change  is 
substantial.  When  there  is  loose  coupling 
and  engineering  slack,  which  permits  devia¬ 
tion  to  occur  so  that  when  components  are  re¬ 
designed  the  deviation  does  not  influence  the 
other  components,  there  is  less  propensity  to 
redesign.  It  is  argued  that  the  tightness  of 
component  interrelatedness  can  be  traded  off 
against  uncertainty,  and  thus  achieve  more 
effective  control. 

Looking  at  the  contract  life  cycle  as  consist¬ 
ing  of  two  phase,  the  award  and  post-award, 
Martin,  Glover  and  Lenz  (136)  have  demonstrat¬ 
ed  that  program  information  becomes  more  un¬ 
certain  with  the  passage  of  time.  The  first 
phase  of  the  acquisition  cycle  provides  the 
basis  for  target  cost,  whereas  the  second 
phase  with  its  lack  of  order  related  to  pro¬ 
gram  information  contributes  to  cost  growth. 

An  entropy  model  developed  by  Martin  (134)  was 
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FIGURE  11  THE  BASIC  S-CURVE  PATTERN  (35%  SLOPE) 
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applied  to  predict  final  program  cost  as  a 
function  of  target  cost  and  disorder  in  inform¬ 
ation  where:  (See  page  48) 

The  model  was  tested  on  the  SRAM  and  F-5E  air¬ 
craft.  In  the  first  case  the  actual  cost  of 
the  SRAM  development  was  $439  million  compared 
with  the  entropic  cost  model  estimate  of  $456 
million.  In  the  second  case  the  results  for 
the  F-5E  showed  an  estimated  cost  of  10.4  times 
greater  than  the  actual  cost.  However,  a  cost 
growth  of  only  7%  for  the  F-5E  is  unusual  and 
could  account  for  some  of  the  difference.  A 
second  problem  could  have  been  the  uncertainty 
assessment  for  computing  the  entropy  value. 
Because  of  its  potential,  this  approach  of  us¬ 
ing  entropy  to  predict  final  cost  warrants 
continued  investigation. 

Another  measure  system  complexity  which  contri¬ 
butes  to  determining  the  entropy  in  a  system. 
Table  12  illustrates  the  impact  of  complexity 
on  maintainability  and  availability.  Complexi¬ 
ty  is  indicative  of  the  uncertainty  related  to 
potential  disorder  and  resultant  cost  overruns. 
(See  page  48) 

RISK  MODELS 

Many  causal  relations  currently  applied  utilize 
risk,  rather  than  uncertainty  to  predict  pos¬ 
sible  outcomes.  Figure  12  shows  the  relation¬ 
ship  between  risk  and  uncertainty  as  related  to 
causality.  Models  of  known  phenomena  provide 
a  more  certain  basis  for  prediction  than  random 
events  which  are  used  for  estimating  probabi¬ 
lities.  Uncertainty,  on  the  other  hand,  covers 
those  areas  that  are  ill-defined  or  where  there 
is  lack  of  knowledge  of  effects.  (See  page  49) 

This  approach  can  be  extended  to  develop  a  risk 
profile  that  changes  over  time  as  shown  in 
Figure  13  and  Figure  14.  (See  page  49&  50) 

The  paper  by  Admiral  Freeman  shows  this  pro¬ 
file  approach  in  relation  to  the  DSARC  review 
process  as  shown  in  Figure  15.  (See  Page  50) 

Merrow,  Chapel  and  Worthing  (141)  developed  a 
relationship  between  cost  growth  and  "months 
before  initial  operational  delivery"  as  a  func¬ 
tion  of  state-of-the-art.  The  high  state-of- 
the-art  projects  are  most  sensitive  to  cost 
growth  as  shown  in  Figure  16.  (See  page  50) 

The  curves  shown  in  Figures  17  and  18  relate 
the  level  of  technological  uncertainty  to  the 
program  life  cycle  and  advances  in  state-of-the- 
art.  Figure  18  shows  the  probable  impact  of  a 
stretch-out  on  the  technological  uncertainty. 
<See  page  51) 

Figure  19  attempts  to  relate  state-of-the-art 
to  interdependency  and  level  of  concurrency. 

The  likelihood  of  disruption  is  shown  as  a  func¬ 
tion  of  varying  levels  of  concurrency.  The 
more  complex  the  program,  and  the  higher  the 


interdependencies  the  greater  the  likelihood 
of  disruption.  Thus,  the  likelihood  of  dis¬ 
ruption  increases  with  increasing  concurrency. 
(See  page  52) 

Another  version  of  the  likelihood  of  disrup¬ 
tion  is  shown  in  Figure  20  as  a  function  of 
delivery  urgency  and  technological  uncertainty 
under  differing  levels  of  resource  application. 
The  higher  the  level  of  resource  application, 
for  a  given  delivery  urgency  and  technological 
uncertainty,  the  lower  the  likelihood  of  dis¬ 
ruption.  The  number  of  examples  abound  where 
alternate  designs  are  produced  in  parallel  to 
reduce  the  chance  of  failure.  Expanding  capa¬ 
city,  adding  personnel  or  increasing  the  level 
of  testing  and  evaluation  are  other  examples 
of  resource  expenditures  used  to  reduce  uncer¬ 
tainty.  (See  page  52) 

In  regard  to  technological  uncertainty,  Duvi- 
vier  (15)  recommends  the  use  of  technological 
forecasting  to  assess  the  risk  in  meeting  the 
demand  for  increasingly  advanced  technology. 

He  postulates  that  advances  are  extrapolations 
of  current  knowledge  and  that  breakthroughs 
are  rare.  Even  when  breakthroughs  do  occur, 
such  as  the  laser,  it  takes  8  to  12  years  to 
incorporate  them  in  new  systems.  He  shows 
examples  of  engine  weight,  lift  and  fuel  con¬ 
sumption  all  following  smooth  curves.  Thus, 
the  cost  and  benefit  of  new  technologies  can 
be  based  on  an  extrapolation  of  technology 
growth  curves . 

Regression  models  have  contributed  significant¬ 
ly  to  the  under  standing  of  causality  in  the 
acquisition  process.  For  example,  Leech  and 
Earthrowe  (126)  have  shown  that  the  ratio  of 
actual  costs  to  estimated  expenditures  can 
be  predicted  based  on  a  regression  with  actual 
size  of  the  job.  Using  a  sample  of  64  jobs, 
they  developed  a  regression  curve,  where  r  * 
.955  +  .009X,  and  X  =  actual  job  size  in  man 
hours.  As  they  point  out,  in  every  case  a 
commitment  was  made  to  the  customer  based  on 
an  initial  design.  However,  where  the  job  is 
large,  requires  considerable  technical  inno¬ 
vation  and  the  quantity  ordered  is  small  (no 
opportunity  for  learning)  the  design  and  dev¬ 
elopment  costs  contribute  significantly  to 
the  final  cost.  They  recommend  an  investment 
portfolio  approach  to  minimize  the  risk  asso¬ 
ciated  with  design  uncertainty. 

It  is  well  known  that  new  technologies  rarely 
are  used  for  the  entire  system.  Rather,  they 
represent  a  small  percent  of  all  the  subsystem 
and  components.  Where  the  manager  maintains 
control  over  those  components  which  utilize 
new  technology,  they  are  in  a  better  position 
to  effect  the  reduction  of  cost  overruns.  The 
use  of  the  Pareto  Law  as  a  basis  for  determin¬ 
ing  which  components  contribute  most  to  the 
technological  uncertainty  is  shown  in  Figure 
21. 

The  point  C  on  the  absicissa  represents  the 
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0.9 
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36.9 
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28.5 
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39.7 

0.4 

43.5 

A-7E 

medium 

36.7 

0.4 

53.0 

F-4J 

medium 

34.2 

0.3 

82.7 

A-6E 

high 

39.3 

0.3 

71.3 

F-14A 

hi  gh 

47.1 

0.3 

97.8 

TABLE  12  -  COMPLEXITY,  MISSION  CAPABILITY,  AND  MAINTAINABILITY  OF  VARIOUS 
WEAPON  SYSTEMS. 

(Source:  Armed  Forces  Journal  International,  May  1980) 
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FIGURE  12  TAXONOMY  OF  CAUSALITY  ANO  UNCERTAINLY 


FIGURE  13  RELATIONSHIPS  AMONG  COST,  RISK,  UNCERTAINTY 
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FIGURE  14  RISK  PROFILE  (TIME) 
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FIGURE  16  VARIATION  OF  COST  GROWTH  WITH  PROGRAM  DURATION 

AND  LEVEL  OF  TECHNICAL  DIFFICULTY 
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FIGURE  18  IMPACT  OF  STRETCH  OUT 
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subset  of  components  that  contribute  most  to 
technological  uncertainty  in  terms  of  system 
impact.  Typically,  20%  of  the  components  con¬ 
tribute  80%  of  the  problems  encountered.  The 
Pareto  Law  can  be  used  as  a  basis  for  para¬ 
metric  estimating  and  provides  a  useful  tool 
to  control  technological  uncertainty. 

(See  page  54) 

Because  technological  uncertainty  impacts  pro¬ 
jects  with  advanced  state-of-the-art.  Reduc¬ 
tion  in  development  time  is  possible  by  main¬ 
taining  a  strong  Research  and  Development  (R&D) 
posture.  New  technologies  can  be  tested  and 
evaluated  prior  to  incorporation  in  major  sy¬ 
stems  and  thus  "avoid1'  the  uncertainty.  Con¬ 
sidering  that  new  technology  is  limited  to  a 
small  percent  of  components,  advanced  or  anti¬ 
cipatory  development  can  contribute  signifi¬ 
cantly  to  the  reduction  of  technological  un¬ 
certainty,  reduced  need  for  concurrency  and 
ultimately  reduced  disruption.  Thus,  "demon¬ 
strated"  technical  capability  could  supplement 
fjy  before  buy  as  an  approach  to  the  manage¬ 
ment  of  risk  and  uncertainty  in  major  ac¬ 
quisitions  . 

This  latter  position  is  consistent  with  DoD 
5000.3,  dated  April  1978  which  states,  "Test 
and  evaluation  shall  be  commenced  as  early  as 
possible  and  conducted  throughout  the  system 
acquisition  process  as  necessary  to  assess 
and  reduce  the  acquisition  risk".  It  also 
concurs  with  OMB-A-109  which  states,  "When 
risks  can  be  accommodated  and  progress  indi¬ 
cates  that  a  proof  of  concept  demonstration  is 
in  order,  the  alternative  system  design  con¬ 
cepts  selected  for  consideration  for  competi¬ 
tive  demonstration  are  to  be  submitted  to  the 
agency  head  for  approval,  along  with  other 
alternatives  which  were  identified  and  evalu¬ 
ated."  Although  early  prototyping  offers  a 
number  of  advantages,  the  maintenance  of  a 
basic  technological  capability  consistent 
with  emerging  needs  can  effectively  collapse 
the  time  span  taken  for  major  developments. 

In  the  commercial  field  IBM  and  Bell  labs  are 
examples  of  maintaining  continuous,  high 
technology,  R&D  capability  which  has  payoff 
in  terms  of  capability  in  developing  new  tech¬ 
nology  . 

CURRENT  APPROACHES  TO  ACQUISITION  MODELING 

This  portion  of  the  report  will  examine  repre¬ 
sentative  models  that  are  currently  being 
applied  to  the  acquisition  of  major  systems. 

For  our  purposes  the  types  of  models  will  be 
grouped  into  two  major  categories  -  probabi¬ 
listic/stochastic  models  and  general  models. 
Within  this  framework  several  aspects  of  each 
of  the  models  will  be  explored  -  namely,  the 
basic  approach  of  the  model,  how  it  is  used, 
results  of  its  use  (post-mortem  analysis  will 
be  included  where  available),  the  requirements 
for  its  use,  and,  the  problems  or  limitations. 
The  selection  of  the  models  chosen  for  analy¬ 
sis  is  not  intended  to  be  exhaustive,  but 
rather  indicative  of  the  type  of  models 


currently  being  used  or  proposed  for  use  in 
the  acquisition  process. 

The  extension  of  the  two  groups  of  models 
leads  to  a  proposed  approach  -  the  Causal  -  In¬ 
tegrative  Model  (CIM)  -  which  is  suggested  as 
a  means  to  deal  with  factors  beyond  those  used 
by  many  of  the  current  models. 

Stochastic/Probabilistic  Models 

Within  this  category,  three  models  will  be  dis¬ 
cussed.  These  are  -  PERT,  VERT,  and  a  Risk 
Analysis  Model  presented  by  Admiral  Freeman 
at  the  1979  Symposium  on  Risk  and  Uncertainty. 

PERT  -  Program  Evaluation  and  Review  Technique 

PERT  was  originally  developed  in  the  late 
1950's  as  a  tool  for  managing  large,  complex 
engineering  projects.  Among  the  charnel er I s- 
tlcs  of  PKKT  ns  a  program  management  tech¬ 
nique  are : 

•  provides  management  with  probabilistic 
estimates  for  best  possible  use  of  re¬ 
sources  to  achieve  a  given  goal  within 
time  and  cost  limitations. 

•  helps  management  examine  the  uncertainties 
involved  in  programs  by  answering  questions 
such  as  how  time  delays  in  certain  elements 
influence  project  completion;  where  slack 
exists;  and  what  elements  are  critical  to 
meet  completion  date. 

•  provides  a  basis  for  evaluation  of  alter¬ 
native  approaches  including  manpower,  mat¬ 
erial,  capital  requirements  through  utili¬ 
zation  of  a  network  analysis. 

A  set  of  basic  requirements  to  meet  the  un¬ 
certainty  aspects  is  as  follows: 

•  a  network  of  events  and  activities  -  Work 
Breakdown  Structure  -  introduced  with  their 
associated  risk  or  uncertainty. 

•  Events  and  activities  are  sequenced  on  the 
network  under  a  highly  logical  set  of  rules 
which  allow  the  determination  of  probabili¬ 
ty  of  delay  or  disruption  on  critical  and 
subcritical  paths. 

The  use  of  PERT  provides  the  program  manager  a 
planning  tool  to  create  a  major  network.  This 
can  be  used  to  analyze  the  interdependencies 
and  problem  areas  which  are  neither  obvious 
nor  well  defined  by  other  techniques.  PERT 
is  also  used  to  estimate  the  probability  of 
meeting  specified  deadlines  by  examining  al¬ 
ternative  plans  and  evaluating  the  effect  of 
changes  in  the  program. 

A  major  disadvantage  of  PERT  is  the  cost  in 
time  and  resources.  Data  collection,  manpower, 
lag  time  to  output,  and  computer  resources, 
are  some  of  the  major  problems.  Maintainance 
of  the  program  is  also  relatively  expensive. 
Thus,  PERT  has  been  primarily  used  on  large, 
complex  systems  where  the  cost  is  a  minor  part 
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of  total  acquisition. 

VERT  -  Venture  Evaluation  and  Review  Technique 

VERT  was  developed  in  1973  by  Gerald  Moeller 
(92)  and  has  been  used  almost  exclusively  by 
U.S.  Army  program  managers  to  determine  the 
"best"  balance  among  the  three  program  para¬ 
meters:  cost,  schedule,  and  performance.  The 

model  evolved  from  earlier  methodological  ap¬ 
proaches  such  as  GERT  (Graphical  Evaluation 
and  Review  technique) ,  CPM  (Critical  Path 
Method),  PERT  (Program  Evaluation  and  Review 
Technique) ,  MATHNET  (Mathematical  Network 
Analyzer),  and  RXSCA  (Risk  Information  System 
and  Cost  Analysis) .  The  short  comings  of 
these  earlier  models  when  compared  with  VERT 
was  their  failure  to  include  the  performance 
variables  along  with  the  cost  and  schedule 
variables  in  the  total  risk-analysis  method¬ 
ology.  The  VERT  model  corrects  this  problem. 

The  VERT  approach  is  a  general  networking 
method  that  determines  program  risk  analysis 
through  two  basic  steps:  construction  of  the 
graphically  representative  network  and  analysis 
of  the  network  using  the  VERT  software.  The 
first  step  entails  development  of  the  ordered 
series  of  activities  or  subtasks  that  lead  to 
a  specific  task.  This  network  includes  all 
aspects  -  including  decision  points  -  required 
to  complete  the  event.  If  the  problem  is 
quite  large  and  complex,  lower  level  networks 
or  subnetworks  of  the  major  subsystems  are 
developed. 

Once  the  network  (or  networks)  is  developed, 
the  program  is  converted  into  the  VERT  soft¬ 
ware  compatible  terminology.  The  software  al¬ 
lows  for  a  variety  of  input  capabilities  that 
make  it  possible  for  decision  events  and  acti¬ 
vities  occurring  within  the  network  to  be  des¬ 
cribed.  Numerical  values  for  a  task's  time, 
cost,  and  performance  are  assigned  along  with 
probabilities  or  decision  rules  based  on  a 
specified  relationship.  The  process  involves 
a  Monte  Carlo  simulation  in  which  the  design 
of  the  network  flow  across  the  entire  network 
or  subnetworks  from  the  start  to  an  appropriate 
decision  point  leads  to  a  trial  solution  of  the 
problem  being  modeled. 

The  process  is  iterated  as  many  times  as  the 
need  warrants  in  order  to  create  a  large  sample 
of  possible  outcomes  concerning:  slack  time, 
completion  time,  cost,  and  performance.  Fre¬ 
quency  distributions,  scatter  diagrams,  and 
probabilities  of  exceeding  given  values  are 
also  generated.  Finally,  pictorial  histogram 
are  generated  for  desired  events,  giving  the 
program  manager  an  integrated  risk  analysis 
for  a  particular  point  of  interest  in  the  pro¬ 
gram.  Mann  (131)  reported  in  Defense  Manage¬ 
ment  Journal  that  "some  minor  problems  have 
arisen  with  VERT,  but  none  are  considered  major 
obstacles  to  its  effective  use."  The  problems 
center  about  the  probability  distributions. 


Most  data  sets  in  VERT  are  triangular  indicat¬ 
ing  pessimistic,  optimistic,  and  most  likely 
values.  This  factor  reduces  the  flexibility 
of  the  model  and  the  accuracy  of  the  simula¬ 
tions.  Another  problem,  according  to  Mann,  is 
the  inability  to  obtain  expert  estimates  of 
the  time  and  cost  requirements.  The  experi¬ 
ence  is  that  most  of  the  values  obtained  have 
been  overly  optimistic  -  which  reduces  the 
usefulness  of  this  approach. 

General  Models 

Within  this  category,  types  of  models  will  be 
discussed  -  parametric  cost  estimation  and 
dynamic  modeling. 

Parametric  Cost  Estimation 

Parametric  Cost  Estimation  is  the  primary 
costing  methodology  for  DoD  weapon  system 
acquisition.  This  approach  evolved  from  re¬ 
search  by  RAND  Corporation  in  the  late  1950’s. 
The  basic  idea  was  to  make  accurate  estimates 
of  weapon  system  costs  at  the  early  stages  of 
system  design.  This  approach  uses  performance 
variables  such  as  speed,  weight,  range,  power, 
etc.  to  predict  costs  since  estimates  of  these 
parameters  are  usually  known  early  in  the  de¬ 
sign  phase.  These  estimates  are  based  on 
historical  data  of  previous  or  similar  systems 
and  utilize  statistical  relationships  between 
cost  and  the  performance  parameters  of  these 
past  or  similar  systems. 

These  statistical  relationships,  called  cost 
estimating  relationships  (CER) ,  take  the  form 
of  an  equation  using  cost  as  a  function  the 
performance  variables  and  constant  coeffici¬ 
ents.  McNichols  (138)  describes  the  relations 
in  simplified  format  by: 

C  -  f  (X)  -  f  (Xr  x2 . Xn  ) 

where  x^  denotes,  a  performance  parameter. 

The  total  cost  would  then  depend  on  each  of 
the  values  of  Xj_  based  on  data  from  historical 
or  similar  systems.  McNichols  criteria  for 
selection  of  the  variables  is  given  by: 

•  The  logical  or  theoretical  relation  of  a 
variable  to  cost  (thus  implying  that  a 
real  dependence  between  cost  and  the  value 
of  the  particular  variable  or  set  of  vari¬ 
ables  exists,  subject  to  some  random  dis¬ 
turbance  or  uncertainty.) 

•  The  statistical  significance  of  the  vari¬ 
able's  contribution  to  the  explanation  of 
cost  (thus  implying  that  relevant  cost  ex¬ 
perience  exists  to  test  and  calibrate  the 
postulated  cost  dependence  -  subject  to 
measurement  uncertainty.) 

•  The  dependence  pattern  of  the  contribution 
made  by  a  variable  to  the  explanation  of 
cost  (thus  the  analyst  must  have  suffi¬ 
cient  confidence  in  the  relationship  that 
he  is  willing  to  extend  it  to  estimate  a 
new  item  -  and  different  analysts  will 
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have  different  degrees  of  confidence  hence). 

There  are  several  advantages  to  the  parametric 
cost  estimation  approach.  First,  since  the 
method  consists  of  a  series  of  CER,s  and  re¬ 
quires  aggregation,  it  is  easily  adapted  to  a 
computer.  Output  and  turnaround  for  new  esti¬ 
mates  can  be  obtained  quickly  when  compared 
with  the  detailed  engineering  approach.  Sec¬ 
ond,  sensitivity  analysis  is  easily  performed 
using  this  method.  For  any  change  in  a  given 
parameter,  t  ho  correspond  tug  change  in  cost 
is  easily  determined.  Third,  cost/benefit 
analysis  or  trade-offs  are  also  easy  to  per¬ 
form.  Fourth,  each  time  a  later  generation 
system  is  estimated,  the  historical  data  base 
already  developed  can  be  updated  and  used. 

The  approach  is  not  without  its  disadvantages. 
First,  the  cost  of  computer  resources  is  sig¬ 
nificant.  Collection  of  the  data  is  time-con¬ 
suming  as  well  as  subjective.  Second,  keeping 
the  database  relevant  is  a  major  problem. 

Haese  (117)  states  that  the  tremendous  techno¬ 
logical  advances  of  weapon  system  state-of-the 
-art  have  tended  to  out-date  cost  data  even 
before  it  is  reported.  Thus,  cost  data  col¬ 
lected  on  the  latest  weapon  system  may  not  re¬ 
present  the  cost  of  current  technology.  With 
changes  from  discrete  components  to  inte¬ 
grated  circuits,  from  compound  metals  to  com¬ 
posite  materials,  etc.  What,  if  any,  histo¬ 
rical  technology  is  similar  enough  to  any  pro¬ 
posed  weapon  system  to  allow  valid  design  and 
credible  cost  comparisons?  Third,  the  rele¬ 
vance  of  the  cost  data  base  is  equally  influ¬ 
enced  by  differences  in  weapon  system  acqui¬ 
sition  management  philosophies,  contractual 
approaches,  contract  types,  and  resources  a- 
vailable.  Fourth,  the  comparability  of  the 
cost  data  among  contractor  generated  cost 
reports  produces  serious  problems.  Often,  it 
is  difficult  to  understand  what  the  collected 
cost  data  represents. 

Risk  Analysis  Model 

RADM  Freeman's  risk  analysis  model  allows  var¬ 
ious  alternatives  or  systems  to  be  objectively 
compared  through  aggregate  risk  analysis.  The 
process  begins  with  a  segmentation  of  the  var¬ 
ious  program  functions  into  categories  of  re¬ 
flecting  the  schedule,  cost,  and  performance 
variables.  Risk  distributions,  represented 
by  utility  functions,  are  used  to  determine 
utility  values  versus  a  change  in  one  of  the 
variables.  For  example,  the  question  of  "how 
much  additional  risk  is  presented  by  a  change 
in  performance  variable  A?"  is  answered.  The 
next  step  consists  of  developing  a  Risk  Matrix 
where  the  options  (or  alternative  systems)  are 
presented  versus  the  criteria  for  choice.  The 
summ.uv  risk  or  probability  for  each  system/ 

:-t  1 1  orintt  l  ve  can  then  he  compared  on  a  quanti- 
l  at  tw  basis. 

The  term  risk  factor  is  presented  in  the  form 


of  an  equation: 

Rf  =  1  -  Ps  (1  -  Cf) 

Where:  Rf  -  Risk  Factor 

Ps  *  Probability  of  Success 
Cf  =  Consequences  of  Failure 
With:  0  <_  Ps  <_  1 

0  <  Cf  <  1 

If  Cf  the  consequence  of  failure  is  interpret¬ 
ed  to  represent  a  utility  function,  then  the 
risk  factor  curve  will  be  defined  as  a  utility 
function.  The  shape  of  this  function  will  be 
in  the  form  of  Figure  20,  previously  described 
If  the  system  criteria  and  associated  risks 
developed  from  the  Risk  Matrix  earlier  in  the 
sequence  were  plotted  in  rank-ordered  fashion, 
it  too  would  be  representative  of  Figure  20 
(or,  a  negative  Pareto  function). 

Dynamic  Modeling 

Computer-based  dynamic  modeling  was  developed 
by  J.W.  Forester  in  the  1950's  as  an  approach 
in  helping  to  solve  problems  of  complex,  con¬ 
tinuous  systems.  In  his  book,  Industrial  Dy¬ 
namics  (105),  Forrester  states  that  a  dynamic 
model  of  a  system  should  have  the  following 
characteristics : 

•  A  statement  of  cause-effect  relationship 

.  Simple  mathematical  relationships 

.  Be  extendable  to  large  numbers  of  vari¬ 
ables  without  exceeding  the  computer  limit¬ 
ations  : 

•  Be  able  to  handle  "continuous"  interac¬ 
tions  in  the  sense  that  any  artificial 
discontinuities  introduced  by  solution¬ 
time  intervals  will  not  affect  the  re¬ 
sults.  It  should,  however,  be  able  to 
generate  discontinuous  changes  in  deci¬ 
sions  when  these  are  needed. 

A  dynamic  model  is  based  on  four  factors  that 
have  improved  understanding  of  complex  systems. 

•  The  theory  of  information-feedback  systems. 

•  A  knowledge  of  decision-making  processes. 

•  The  experimental  approach  to  analysis  of 
complex  systems. 

•  The  digital  computer  as  a  means  to  simu¬ 
late  realistic  mathematical  models. 

Forrester  contends  that  the  development  of  a 
dynamic  model  should  encompass  the  following 
steps : 

•  Identify  a  problem 

•  Isolate  the  factors  that  appears  to  inter¬ 
act  to  create  the  observed  symptoms 

•  Trace  the  cause-and-eff ect  information- 
feedback  loops  that  link  decisions  to 
action  to  resulting  information  changes 
and  to  new  decisions 

•  Formulate  acceptable  formal  decision 
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policies  that  describe  how  decisions  result 
from  the  available  information  streams 

•  Construct  a  mathematical  model  of  the  deci¬ 
sion  policies,  information  sources,  and  in¬ 
teractions  of  the  system  components 

•  Generate  the  behavior  through  time  of  the 
system  as  described  by  the  model 

•  Compare  results  against  all  pertinent  a- 
vailable  knowledge  about  the  actual  system 

•  Revise  the  model  until  it  is  acceptable  as 
a  representation  of  the  actual  system 

•  Redesign,  within  the  model,  the  organiza¬ 
tion  relationships  and  policies  which  can 
be  altered  in  the  actual  system  to  find  the 
changes  which  improve  system  behavior 

•  Alter  the  real  system  in  the  directions 
that  model  experimentation  has  shown  will 
lead  to  improved  performance 

He  further  states  that  dynamic  models  should  be 
based  on  these  premises: 

•  Decisions  in  management  and  economics  take 
place  in  a  frame  work  that  belongs  to  the 
general  class  known  as  information- feed¬ 
back  systems. 

•  Intuititive  judgment  is  unreliable  about 
how  these  systems  will  change  with  time, 
even  when  good  knowledge  of  the  individual 
parts  of  the  system  is  possessed. 

•  Model  experimentation  is  now  possible  to 
fill  the  gap  where  judgment  and  knowledge 
are  weakest  -  by  showing  the  way  in  which 
the  known  separate  system  parts  interact 
to  produce  unexpected  and  troublesome  over¬ 
all  system  results. 

•  Enough  information  is  available  for  this 
experimental  model-building  approach  with¬ 
out  great  expense  and  delay  in  further 
data  gathering. 

•  The  "mechanistic"  view  of  decision  making 
implied  by  such  model  experiments  is  true 
enough  so  that  the  main  structure  of  con¬ 
trolling  policies  and  decision  streams  of 
an  organization  can  be  represented. 

•  Industrial  systems  are  constructed  inter¬ 
nally  in  such  a  way  that  they  create  for 
themselves  many  of  the  troubles  that  are 
often  attributed  to  outside  and  independ¬ 
ent  causes. 

•  Policy  and  structure  changes  are  feasible 
that  will  produce  substantial  improvement 
in  industrial  and  economic  behavior;  and 
system  performance  is  often  so  far  from 
what  it  can  be  that  initial  system  design 
changes  can  improve  what  causes  losses  in 
one  area  in  exchange  for  gains  in  another. 

Dynamic  system  model  contain  four  essential 
features : 

•  Several  levels  or  accumulations  within  the 
systems;  e.g.  -  inventories;  number  of  em¬ 
ployees,  work  in  process. 

•  Flows  rates  that  transport  the  contents  of 
one  level  to  another. 

•  Decision  functions  that  control  the  rates 


of  flow  between  levels. 

•  Information  channels  that  connect  the  de¬ 
cision  function  to  the  levels  concerned. 

Figure  22  depicts  an  example  of  the  interaction 
of  the  major  components  of  a  dynamic  system. 

In  Figure  22,  the  solid  line  represent  the  flow 
channels  and  the  dashed  lines  represent  inform¬ 
ation  sources.  Not  shown  in  Figure  22  are  all 
of  the  associated  delays  with  the  various  lev¬ 
els  (shown  is  the  delivery  delay).  For  example, 
delays  associated  with  hiring  and  training  em¬ 
ployees,  manufacturing  delays,  delays  in  re¬ 
ceiving  raw  materials,  and  production  delays 
associated  with  the  various  levels  and  flows. 

.(Shown  on  page  54) 

Expansion  of  the  concepts  presented  by  Forres¬ 
ter  into  an  acquisition  model  would  present  a 
clear  picture  of  possible  problems,  areas  of 
likelihood  of  cost  overrun,  and  disruptions. 
Thus,  the  main  advantage  of  dynamic  simulation 
is  that  it  forces  managers  to  crystalize  their 
decision-making  processes.  This  function  can¬ 
not  help  but  lead  to  greater  insights  into 
program  acquisition. 

However,  the  approach  is  not  without  disadvan¬ 
tages.  Among  these  are: 

.  In  simulation,  all  relevant  variables  and 
phenomena  must  be  quantified.  The  reduc¬ 
tion  of  all  descriptive  knowledge  to  quant¬ 
itative  measures  is  not  always  valid. 

.  Dynamic  simulation  is  found  to  be  most  use¬ 
ful  in  price-quantity  problems,  less  useful 
in  organizational  design,  and  least-useful 
in  product-market  strategy. 

.  Dynamic  simulation  is  not  easy  to  apply. 

It  is  a  complex  technique  that  needs  con¬ 
siderable  data  and  knowledgeable  people. 

.  There  are  problems  in  acceptance  of  the 
approach  because  it  is  often  considered 
a  research  tool. 

Causal  Integrative  Model  (CIM) 

An  extension  of  the  Forrester  type  model  is 
shown  in  Figure  22  which  describes  the  pro¬ 
cesses,  flows,  variables,  feedback  loops,  de¬ 
lays,  exogenous  variables  and  key  decisions  as 
they  are  related  to  the  four  basic  variables 
in  the  acquisition  process.  As  noted  in  the 
report,  acquisition  models  currently  being 
used  do  not  address  all  of  these  variables, 
thus,  each  model  lacks  some  degree  of  com¬ 
pleteness  . 

Referring  to  Figure  23,  the  Causal-Integrative 
Model  can  be  used  to  determined  how  a  change 
in  economic  uncertainty  affects  the  level  of 
environmental  uncertainty  which,  in  turn,  af¬ 
fects  mission,  scope,  and  funding.  These 
changes  pertubate  the  system  to  effect  changes 
in  organizational  slack,  technological  uncer¬ 
tainty,  and  customer  urgency.  Thus,  a  change 
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in  one  variable  can  be  shown  to  cause  changes 
in  the  others  through  the  pervasive  network  of 
interdependencies.  These  changes  in  the  key 
variable  impact  the  acquisition  cycle  in  ways 
that  are  not  intuitively  obvious  without  the 
aid  of  a  dynamic  model  to  point  out  the  causal 
relationships . 

The  direction  in  acquisition  management  promp¬ 
ted  by  this  report  leaves  several  facets  to  be 
completed : 

.  development  of  a  computer-based  model  for 
use,  such  as  the  CIM, 

.  testing  of  the  model  with  a  completed  pro¬ 
gram, 

.  validation  of  the  model  using  current  pro¬ 
grams  , 

.  implementing  the  model  for  policy  level 
decisions  in  acquisition  management. 

In  order  to  extend  the  research  described  a- 
bove,  additional  research  is  needed  to  con¬ 
struct  the  Causal-Integrative  Model. 

CONCLUSION 

The  material  presented  in  this  report  has  at¬ 
tempted  to  highlight  Important  advances  that 
have  been  made  in  improving  the  acquisition 
process.  Because  of  the  pervasiveness  of  the 
subject,  of  necessity,  not  all  relevant  re¬ 
search  of  applications  can  be  included.  Rather, 
what  has  been  presented  here  can  be  considered 
as  indicative  of  the  current  state-of-the-art 
in  acquisition  management  and  a  baseline  ap¬ 
proach  for  future  developments. 

At  the  outset,  the  report  emphasized  the  need 
for  a  causal  basis  for  understanding  the  fac¬ 
tors  that  affect  cost  overruns.  A  number  of 
illustrations  were  presented  that  clearly  iden¬ 
tify  that  cost  growth  is  a  phenomena  that  is 
related  to  the  acquisition  of  complex  projects, 
both  civilian  and  military.  Furthermore,  that 
four  primary  variables  contribute  to  cost. 

These  include  environmental  uncertainty,  tech¬ 
nological  uncertainty,  customer  urgency  and 
organizational  slack.  The  discussion  pointed 
out  that  control  as  currently  practiced,  is 
not  sufficient  to  avoid  cost  overruns.  A  num¬ 
ber  of  research  reports  were  reviewed  which 
cover  reasons  for  cost  overruns,  including  an 
extensive  study  done  at  the  University  of 
Southern  California  which  identified  26  factors 
contributing  to  cost  overrun.  Among  the  key 
contributors  to  incurring  higher  than  budgeted 
costs  are  the  four  primary  variables  in  the 
acquisition  process  described  above. 

Having  established  a  basis  for  understanding 
why  cost  overruns  occur,  the  next  section  of 
the  report  dealt  with  the  risk  and  uncertainty 
aspects  of  the  problem.  A  number  of  authors 
were  cited  to  help  define  the  sometimes  ab¬ 
struse  terminology  used  in  the  field.  This 
material  provided  a  foundation  for  the  sec¬ 
tion  on  a  causal  basis  for  defining  uncertainty 


A  number  of  studies  were  presented  to  help  un¬ 
derstand  what  causes  uncertainty  and  how  to  ap¬ 
proach  it  in  the  acquisition  process.  For  ex¬ 
ample,  it  was  pointed  out  that  uncertainty  and 
disruption  cannot  be  elinimated,  but  rather  can 
be  controlled  if  there  are  causal  models  such 
as  the  RANT)  study  relating  cost  to  advance  in 
state-of-the-art.  Each  of  the  four  basic  vari¬ 
ables  were  examined  in  terms  of  developing  mo¬ 
dels,  measures  or  approaches  to  coping  with  the 
problems  of  uncertainty. 

Given  the  foundation  presented  to  this  point,  a 
set  of  causal  relations  among  variables  in  dis¬ 
ruption  and  uncertainty  were  examined  in  order 
to  establish  a  "pattern  of  disruption".  This 
was  followed  by  the  section  on  current  ap¬ 
proaches  to  acquisition  modeling,  including 
ones  used  for  risk  analysis. 

The  final  section  presented  a  "Causal-Integra¬ 
tive  Model",  which  illustrates  the  complex  re¬ 
lationships  that  exist  among  the  variables  that 
affect  the  acquisition  process.  Although  this 
is  a  preliminary  model,  it  provides  a  basis 
for  integrating  the  approaches  to  date  to  mana¬ 
ging  the  acquisition  process.  It  includes  many 
causal  sub-models,  such  as  concurrency,  learn¬ 
ing  curve,  disruption,  etc.  It  also  covers  the 
dynamic  interdependencies  that  exist  and  the 
treatment  of  risk  and  uncertainty  as  integral 
parts  of  the  model. 

Acquisition  managers  armed  with  more  sophisti¬ 
cated  tools  can  improve  the  effectiveness  of 
their  performance  and  thereby  achieve  the  max¬ 
imum  potential  cost  control.  Obviously,  no  set 
of  tools  or  techniques  is  a  substitute  for  man¬ 
agement;  however,  the  well  informed  manager  can 
make  decisions  which  have  a  greater  likelihood 
of  occurrence  providing  the  bases  used  are 
sounder.  The  causal  approach  described  in  this 
report  offers  the  potential  for  achieving  this 
goal. 
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APPENDIX 

As  an  adjunct  to  the  research  report,  a  survey 
of  various  government  agencies,  military  ser¬ 
vices,  contractors,  and  university  faculty  was 
conducted.  The  goal  of  the  survey  was  to  sum¬ 
marize  methods  currently  being  used  or  proposed 
to  be  used  in  the  analysis  of  risk  and  uncer¬ 
tainty  in  major  program  acquisitions.  As  ex¬ 
pected,  many  of  the  approaches  relied  on  DoD 
Directives  5000.1  and  5000.2  along  with  DMB 
Circular  A-109. 

The  following  comments  were  either  excerpted 
directly  from  the  returned  questionnaire  or 
were  summarized  by  the  authors. 

.  This  respondent  is  not  involved  in  the  ac¬ 
quisition  process,  but  does  perform  econo¬ 
mic  analyses  in  support  of  it.  Encouraged 
in  the  use  of  sensitivity  analyses  or  "what 
if"  types  of  approaches.  The  prime  tool  is 
the  net  present  value  approach  (NPV)  of 
different  investments  using  inputs  concern¬ 
ing  major  uncertainties.  If  changing  the 
risk  changes  the  rankings  of  the  NPV  anal¬ 
ysis,  the  factor  is  considered  sensitive 
for  subsequent  analyses. 

Also  included  from  the  respondent  was  a 
copy  of  proposed  legislation  HR  8303  dated 
October  2,  1980.  This  bill  would  establish 
coordinated  program  under  the  direction  of 
the  Office  of  Science  and  Technology  Policy 
for  improving  and  facilitating  the  use  of 
risk  analysis  by  those  Federal  Agencies 
concerned  with  scientific  and  technological 
decisions  as  related  to  human  life,  health, 
and  protection  of  the  environment.  The 
bill  would  require  that  research  be  done 
to  meet  the  objectives  defined  within  the 
bill.  These  objectives  deal  with  establish¬ 
ing  criteria  and  methods  for  risk  analysis. 

.  Several  different  techniques  are  used  by 
this  group.  These  include: 

.  establishing  management  reserves  for 
contingencies 

.  cost  and  schedule  variance  reporting 
using  C/SCSC  (Cost/Schedule  Control 
System  Criteria) 

.  use  of  program  "baselines"  which  estab¬ 
lish  program  cost-levels  to  carry  out 
the  program 

Noted  that  formal  approaches  on  risk  and 
uncertainty  are  limited  in  usage  within  DoD 

.  A  bidding  division  provides  the  corporate 
offices  with  details  of  the  customer’s 
request  for  proposal  (RFP)  for  Independent 
review  and  comment.  Discussions  ensue  on 
high  risk  areas  on  how  to  minimize  the  risk 
through  contract  terms  or  price  considera¬ 
tions.  Corporate  level  approval  Is  ob¬ 
tained  via  a  proposal/pricing  review  meet¬ 
ing  where  risks  and  treatments  of  risk  are 


discussed.  Corporate  approval  usually 
carries  certain  qualifying  guidelines. 

Any  departures  from  these  guidelines  dur¬ 
ing  competition  must  be  reviewed  and  ap¬ 
proved. 

Risk  assessment  (reduction?)  is  considered 
and  incorporated  as  follows: 

.  Need  for  a  validation  period  for  criti¬ 
cal  components /segment 
.  Need  to  maintain  multiple  sources 
through  the  validation  plan,  and 
through  full  scale  development  if  risks 
are  sufficiently  low 
.  Extent  of  testing  (number  and  types) 
with  rules  for  operational  evaluation 
testing  being  employed 
.  Extent  of  testing  and  demonstration 
prior  to  commiting  to  production 
.  Type  of  contract  (cost  vs.  fixed  price) 
and  type  of  incentive  to  recognize  risk 
and  drive  contractual  efforts  towards 
technical  solution 
Mainly  employs  Dod  5000  series. 

Describes  a  general  management  approach 
for  addressing  Dod’s  affordability  pro¬ 
blem  in  the  procurement  funding  area  LMI 
report  RE903  "Affordability  for  Major 
Systems  Acquisitions"  Jan  1980.  The  af¬ 
fordability  analysis  procedure  is  based 
on  the  use  of  an  affordability  matrix 
for  the  procurement  analysis  of  each  ser¬ 
vice.  The  matrix  allocates  forecasted 
procurement  resources  to  programs  for  a 
15-year  period  based  on  program  priority 
and  cost,  thereby  establishing  a  15-year 
baseline  procurement  program  for  each 
service.  This  procedure  could  lead  to 
stable  funding  for  high  priority  programs 
and  identification  of  unaffordable  pro¬ 
curement  programs. 

Computer-based  approach  that  used  a 
prior  established  upper  and  lower  bounds 
costs  for  subsystem.  (eg.  airframe  en¬ 
gine  control  &  guidance,  etc.).  A  beta 
distribution  is  fitted  to  the  upper,  low¬ 
er,  and  most  likely  estimates  of  costs. 
Subsystem  costs  are  then  aggregated  us¬ 
ing  Monte  Carlo  Simulation. 

Study  under  way  (Study  completion  date  in 
1981) 

DoD  funded  study  involves  interviews  with 
strategic  planners  and  procurement  mana¬ 
gers  regarding  the  process  and  content  of 
strategic  planning  efforts  by  prime  con¬ 
tractors  to  identify  sources  of  material, 
product  and  technological  risk  among  low¬ 
er  tiers  of  the  industrial  base  and  dev¬ 
elop  proactive ,  entrepreneurial  programs 
to  help  reduce  that  risk.  Desired  output 
is  a  viable  and  responsive  multi-tier  In¬ 
dustrial  base  for  major  problems.  Speci¬ 
fic  risk  issues  involve  critical  materials 
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and  capacities,  manufacturing  technologies, 
and  interorganizational  technology  transfer. 

Developed  a  method  of  first  order  evalua¬ 
tion  of  a  system  cost  which  is  applicable 
in  the  first  months  of  a  concept  develop¬ 
ment  effort.  The  method  identifies  the 
major  contributors  to  cost  and  the  factors 
that  influence  them.  Reduction  of  cost 
risk  requires  knowledge  of  where  the  risks 
are.  This  approach  developed  by  the  res¬ 
pondent  provides  the  identification  of  the 
risks  within  "reaction  time".  Utilizes  en¬ 
gineering  estimates  (vs.  parametric). 

Offers  two  methods:  a)  parametric  cost 
estimation  technique,  and  b)  statistical 
sampling. 

The  parametric  estimate  utilizes  "degree 
of  analogy"  as  measured  by  Mahalanobis 
distance  (defined  as  the  distance  between 
two  multivariate  populations.  The  two 
populations  are  the  predictor  variable 
space  and  the  historical  variable  space). 
The  approach  selects  a  data  base  and  ap¬ 
propriate  predictor  variables  to  predict 
the  cost  of  a  major  program  based  on  his¬ 
torical  costs  of  related  programs. 

The  statistical  sampling  approach  is  con¬ 
strained  to  sole  source  pricing.  It  uses 
a  suitably  selected  sample  of  proposals 
from  a  sole  source  to  price  out  an  entire 
backlog.  It  is  shown  that  classical  stat¬ 
istical  procedures  are  vulnerable  to  game- 
mans  hip  and  this  new  approach  is  invulner¬ 
able  to  this  since  the  user  controls  the 
risk  with  sample  size  and  sample  stratifi¬ 
cation. 

Preliminary  study  that  analyzes  acquisi¬ 
tion  costs  as  a  function  of  production 
rate  and  the  quantity  produced. 

Appears  to  pick  up  cost  growth  as  related 
to  customer  urgency  (production  rate)  and 
scope  changes  (quantity  produced) . 

Program  using  cost,  technical,  and  sched¬ 
ule  analyses  in  support  of  development 
and  full  production  of  a  battlefield 
system. 

In-house  and  external  agencies  perform  an¬ 
alyses  at  prescribed  stages  in  the  acquisi¬ 
tion  cycle  (See  Dodd  5000.1,. 2).  These 
analyses  use  the  appropriate- anlytical 
tools  for  the  stated  study  objective  in¬ 
cluding  C/SCSC  (Cost/Schedule  Control  Sy¬ 
stem  Criteria) ,  VERT  (Venture  Evaluation 
and  Review  Technique  -  a  quantitative 
"CPM"  -  type  of  approach)  and  other  opera¬ 
tions  research  systems  analyses  techniques. 

Uses  an  approach  called  PROMAP.  This  ap¬ 
proach  is  a  network  analysis  technique 


similar  to  PERT  and  CPM  but  is  probabilis¬ 
tic  rather  than  deterministic  like  CPM  and 
PERT.  PROMAP  is  a  simulation  program  to 
aid  in  decision-making  regarding  risk  in 
the  areas  of  cost  and  schedule  (does  not 
look  at  technical  risk  or  address  causali- 
ty). 

The  methodology  is  being  used  to  reduce 
risk  and  uncertainty  on  program  costs  and 
schedules  (again,  lacks  technical  risk) 
through  application  of  the  Draft  RFP .  Po¬ 
tential  contractors  are  encouraged  to  chal¬ 
lenge  requirements  that  are  considered  sig¬ 
nificant  "cost  drivers"  and  to  suggest  re¬ 
visions  to  performance,  schedule,  or  other 
contractual  requirements  that  could  en¬ 
hance  program/project  performance. 

In  addition  to  the  DRFP ,  Business  strategy 
panels  are  convened  prior  to  the  issuance 
of  a  formal  RFP.  The  panel  discusses  ac¬ 
quisition  problems,  desired  type  of  con¬ 
tract  (FPI ,  CPFF,  etc.)  and  other  relevant 
aspects  to  this  point  in  the  cycle. 

Based  on  overall  risk  assessment,  the  type 
of  contract  is  determined  (incentives,  fee 
sharing,  etc.)  which  may  be  influenced  to 
some  degree  by  the  contractor. 

Uses  C/SCSC  which  required  detailed  plan¬ 
ning  of  work  tasks  by  cognizant  people  at 
the  operational  level  in  terms  of  antici¬ 
pated  resources  to  be  anticipated  to  be 
expended  for  successful  completion  (C.O.P. 
cost  to  complete) . 

Looks  at  variance  of  cost  and  schedule  and 
performance  against  the  current  plan.  I- 
dentifies  causes  of  variances  to  the  plan, 
impacts  of  variances,  recommends  action, 
and  probable  new  plan.  Manages  risk  on 
an  exception  basis.. 

Because  of  the  variety  of  programs,  res¬ 
pondent  indicates  that  no  standard  ap¬ 
proach  is  desirable.  Instead,  each  pro¬ 
gram  is  handled  as  a  case-by-case  approach 
where  management  reviews  the  risk  and  un¬ 
certainty  prior  to  the  commitments  and  at 
intervals  throughout  program  life.  The 
intervals  are  pre-established  but  reviews 
can  occur  upon  indication  of  a  potential 
change  in  risk  caused  by  either  internal 
or  external  reasons.  At  critical  points, 
these  reviews  are  made  with  the  inclusion 
of  ad  hoc  groups  of  individuals  not  dir¬ 
ectly  associated  with  the  program. 

The  approach  is  to  consider  uncertainty  u- 
biquitous,  and  to  treat  it  descriptively, 
as  a  characteristic  of  variables  of  inter¬ 
est.  Develops  the  term  "Parametric  Factor 
Evaluation"  (PFE)  as  a  process  of  identi¬ 
fying  variables  of  interest  describing  the 
nature  or  form  of  the  uncertainty  of 
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concern,  and  for  examining  available  in- - 
formation  to  arrive  at  the  required  or  de¬ 
sired  decision. 

.  Manages  risk  and  uncertainty  in  the  acqui¬ 
sition  of  major  programs  by  various  methods 
and  techniques  authorized  by  Defense  Ac¬ 
quisition  Regulations  (DARs).  The  regula¬ 
tions  provide  the  use  of  various  types  of 
contracts  depending  on  the  degree  of  risk 
and  uncertainty  (did  not  state  hov  the  de¬ 
gree  Is  obtained).  In  addition  various 
forms  of  escalat Ion  clauses  are  utilized 
in  long  term  major  contracts  to  cope  with 
uncertainty  and  degree  of  inflation  as 
well  as  labor  and  material  cost  increases. 

.  Risk  and  uncertainty  are  assessed  at  the 
outset  of  every  program.  They  are  judged 
mostly  by  experience  on  previous  programs 
and  how  far  current  technology  would  be  ex¬ 
tended  in  each  approach  considered.  Cost 
and  schedule  risk  are  tied  directly  to  per¬ 
formance  goals  and  difficulty  of  achieve¬ 
ment  vis-a-vis  current  technology. 

.  Describes  three  approaches  currently  being 
used : 

.  SCARA  -  Schedule/cost  Risk  Assessment 
which  is  a  detailed  program  simula¬ 
tion  model  used  to  predict  budget  re¬ 
quirements  . 

.  Estimate  at  completion:  statistical 
curve  fitting  routine  based  on  histo¬ 
rical  program  cost  and  schedule  data, 
used  to  predict  program  budgets  and/or 
schedules . 

.  Parametric  Cost  Estimating:  Models 
are  used  to  attempt  to  accurately  pre¬ 
dict  program  runout  costs. 

.  Describes  a  conceptual  approach  that: 

.  Uses  a  triangular  frequency  distribu¬ 
tion  to  describe  risk  profile  of  each 
major  cost  element 

.  Convolutes  these  individual  risk  pro¬ 
files  into  an  overall  risk  profile  for 
the  program 

.  Uses  the  overall  risk  profile  to  es¬ 
tablish  the  negotiation  position  for 
high  risk  (low  definition)  programs 

.  Manages  program  risk  in  shipbuilding  by: 

.  Providing  for  adequate  design  schedule 
and  iterate  design; 

.  Introducing  shipbuilder  and  contractor 
support  early  in  the  program; 

.  commencing  testing  early  at  land-based 
engineering  facilities; 

.  commencing  detailed  design  before  the 
lead  ship  contract  is  awarded; 

.  using  cost  reimbursement  contracts  in 
initial  stages; 


.  using  a  lead  yard-follow  yard  concept 
with  the  ship  builder  to  ship  builder 
to  contractor  liaisons  relationships 
(in-yard  or  in-facility) 

.  bounding  physical  parameters  for  devel¬ 
oping  items. 

Just  completed  study  resulted  in  this  ap¬ 
proach  to  assess  the  feasibility  of  devel¬ 
oping  an  analytical  model  for  use  in  sel¬ 
ecting  an  acquisition  strategy.  Probabi¬ 
lity  distributions  of  key  parameters  are 
determined  by  comhining  subjective  assess¬ 
ment  with  information  contained  in  an  his¬ 
torical  data  base.  From  these,  the  ex¬ 
pected  result  of  pursuing  a  particular  ac¬ 
quisition  strategy  is  assessed  through 
attributes  addressing  time,  cost,  afforda¬ 
bility,  technical  risk,  etc.,  together 
with  the  uncertainties  associated  with 
each.  Tradeoffs  among  the  varying  attri¬ 
butes  are  accomplished  by  developing  a 
multiattribute  utility  model  tailored  to 
reflect  the  needs  and  constraints  of  a 
particular  program. 

Risk  analysis  program  which  is  a  basic 
Monte  Carlo  simulation  model  with  input 
distributions  from  uniform  to  Weibull. 
Utilizes  methods/techniques  as  established 
by  the  federal  government  in  such  docu¬ 
ments  as: 

.OMB A- 10 9 
.DoDD  5000  Series 
. USAF  7000  Series 
.USAF  AS CP  800-19 

Also  utilize  parametric  analysis,  C/SCSC 

Apply  project  risk  estimates  as  developed 
from  the  company's  experience  in  market¬ 
ing,  designing,  manufacturing  and  adjust¬ 
ing  expected  cash  flows  accordingly  by 
adjusting  discount  rate,  or,  if  possible, 
using  certainty  equivalent  method  of  ad¬ 
justing  cash  flows  directly,  so  as  not  to 
compound  effect  of  risk  adjusted  discount 
rate.  Expected  net  present  values  of 
cash  flows  determines  desirability  of  pro¬ 
ject  under  projected  risk. 

.  Methodology  involves  use  of  Award  Fee 
Type  incentive  for  fixed  price  follow- 
ship  contracts  in  order  to  encourage  con¬ 
tractors  to  finish  contracts  on  time, 
with  high  quality,  within  cost  goals. 

.  A  performance  Measurement  System  is  used 
based  on  the  use  of  earned  value  of  work 
performed.  The  system  uses  a  computer 
data  base  developed  by  each  using  organi¬ 
zation  and  program.  The  data  is  evalua¬ 
ted  for  schedule  and  cost  variances  for 
current  and  ITD  periods  of  performance 
and  at  completion  variance. 

.  In  April  1976,  the  Office  of  Management 
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and  Budget  issued  OMB  Circular  A-109  for 
the  acquisition  of  major  systems.  The  cir¬ 
cular  required  that  each  federal  agancy 
adopt  new  procedures  for  early  and  complete 
evaluation  of  alternatives  identified 
through  the  competitive  process.  One  of 
the  key  objectives  of  the  Circular  is  mana¬ 
ging  risk  and  uncertainty  so  that  it  re¬ 
mains  within  tolerable  levels.  This  ob¬ 
jective  is  met  in  several  ways.  First, 
in  the  development  of  an  acquisition  stra¬ 
tegy,  the  project  manager  must  recognize 
and  make  accommodations  for  risk  and  un¬ 
certainty  that  assures  the  proper  relation¬ 
ship  of  risk  sharing  between  Government 
and  Contractors  and  establish  a  management 
system  for  identifying,  analyzing,  and 
evaluating  risk  throughout  the  acquisition 
process.  Secondly,  A-109  employs  the  use 
of  short-term  parallel  contracts  to  fur¬ 
ther  explore  proposed  alternatives  and 
reduce  the  technical  uncertainties.  Dur¬ 
ing  contract  performance  emphasis  is  pla¬ 
ced  on  the  measures  taken  to  progressively 
reduce  risks.  Finally,  throughout  the  re¬ 
mainder  of  the  process  the  head  of  the  a- 
gency  must  assure  himself  that  the  risks 
have  been  evaluated  and  accommodated  be¬ 
fore  approving  the  continuation  of  the 
acquisition  at  the  key  decisions  points — 
selection  of  alternatives  and  approval  to 
advance  to  test /demons tration ,  commitment 
to  full-scale  development,  and  commitment 
to  full  production,  as  appropriate. 

Within  GSA  this  A-109  approach  is  applied 
to  major  public  huildings  and  ADP/Tele- 
communication  system  acquisitions  whose 
estimated  value  is  $25  million  or  greater. 
Specific  guidance  for  the  acquisition  of 
each  type  of  acquisition  has,  or  is,  in 
he  process  of  being  developed. 
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ABSTRACT 


This  paper  addresses  the  results  of  a  survey 
of  the  available  sources  of  information  con¬ 
cerning  the  behavioral  aspects  of  decision¬ 
making,  uncertainty,  and  risk  analysis  in  the 
Department  of  Defense  (DoD)  acquisition  pro¬ 
cess.  The  authors  were  able  to  determine  that 
very  little  information  exists  that  directly 
addresses  small  group  behavior  as  related  to 
the  subject,  that  there  is  disagreement  con¬ 
cerning  optimizing  group  performance,  and  that 
better  risk  analysis  data  could  be  obtained  if 
the  group  members  were  adequately  trained  and 
held  responsible  for  the  group  decisions. 

INTRODUCTION 


There  is  a  great  deal  of  uncertainty  assoc¬ 
iated  with  each  stage  of  the  acquisition  pro¬ 
cess  within  the  DoD.  This  uncertainty  exists 
primarily  in  the  areas  of  cost,  schedule,  and 
performance.  Inability  to  successfully  fore¬ 
cast  in  these  areas  has  led  to  sometimes 
severe  criticism  of  the  acquisition  process, 
in  many  cases  creating  adverse  public  opinion 
and  a  negative  effect  on  U.S.  defense  capabil¬ 
ities.  Current  DoD  policy  requires  that  these 
uncertainties  be  continually  considered 
throughout  all  phases  of  the  acquisition  pro¬ 
cess  and  that  risks  be  minimized  before  a 
production  decision  is  made.  Department  of 
Defense  Directive  (DODD)  5000.2  states  "The 
program  manager  shall  take  positive  action  to 
continually  assess  program  risk  areas  and  to 
make  or  propose  trade-offs  in  performance, 
cost  and  schedule  to  achieve  the  best  bal¬ 
ance.'*  Also,  "Contract  type  shall  be  consis¬ 
tent  with  the  system  program  characteristics 
and  shall  give  particular  emphasis  to  the 
issues  of  risk  and  uncertainty  to  the  con¬ 
tractor  and  government."  The  uncertainties 
involving  the  technological  unknowns  associ¬ 
ated  with  the  weapons  system  acquisition  pro¬ 
cess  require  the  manager  to  face  more  risk  in 
his  decision-making  than  exists  in  other  areas 
of  management. 

It  is  appropriate  at  this  point  to  define  risk 
as  it  is  used  in  relation  to  the  acquisition 
process.  It  is  "The  probability  that  a 
planned  event  will  not  be  attained  within  con¬ 
straints  (cost,  schedule,  performance)  by 


following  a  specified  course  of  action."  [1] 

Much  work  has  been  done  and  continues  to  be 
done  in  the  area  of  development  of  tools  for 
assessing  risk  and  for  performing  risk  analy¬ 
sis.  However,  there  is  another  very  important 
aspect  of  decision-making  under  conditions  of 
risk  and  uncertainty:  the  behavioral  aspects 
associated  with  the  decision-maker  and  his 
staff  in  the  decision-making  process  under 
conditions  of  risk  and  uncertainty.  Concern 
over  this  aspect  led  the  Air  Force  Business 
Management  Research  Center  to  contract  with 
the  University  of  Dayton  to  conduct  a  penetra¬ 
tion  study  in  this  area . 


STUDY  RESULTS 


The  objective  of  this  study  was  to  conduct  a 
survey  of  the  available  sources  of  information 
in  order  to  identify  sources  of  knowledge  and 
information  about  the  behavioral  aspects  of 
decision-making,  uncertainty,  and  risk  analy¬ 
sis  in  the  Department  of  Defense  acquisition 
process.  Throughout  the  survey,  an  effort  was 
attempted  to  emphasize  those  topics  and 
sources  that  can  be  directly  related  to  the 
risk  analysis  and  decision  framework  of  the 
weapons  system  acquisition  process  as  defined 
in  DODD  5000.1  and  DODD  5000.2.  In  addition, 
the  study  concentrated  specifically  on  those 
topics  and  sources  of  information  that  address 
individual  and  small  group  behavior  in  the  un¬ 
certain  decision  environment. 

As  a  result  of  this  contract  effort,  the  study 
team  also  developed  a  catalog  of  sources  of 
knowledge  and  information  for  use  by  acquisi¬ 
tion  managers  and  their  staffs  to  improve  their 
understanding  of  individual  and  small  group 
behavior  in  the  uncertain  decision  environment. 
This  catalog  is  included  as  a  part  of  the  con¬ 
tract  and  addresses,  in  alphabetical  order  by 
author  and/or  topic,  those  articles  that  may 
be  of  interest  to  acquisition  managers.  In 
each  case,  in  addition  to  the  title,  author, 
and  source  of  the  document,  a  complete 
abstract  is  provided  in  the  catalog. 

The  contractor  utilized  the  Defense  Technical 
Information  Center  and  the  DIALOG  Information 
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Retrieval  Service  from  Lockheed  Information 
Systems  to  provide  listings  and  abstracts  of 
appropriate  documents.  Key  words  used  by  the 
study  team  included  risk,  risk  analysis, 
decision-making,  uncertainty,  small  group 
behavior,  group  decision-making,  etc.,  as 
entering  arguments  into  the  data  bases.  The 
remainder  of  this  paper  considers  some  of  the 
more  important  observations  derived  from  the 
literature  review. 

It  was  determined  that  in  almost  all  cases  in 
the  weapons  system  acquisition  process  the 
opinion  of  experts  based  upon  their  experience 
is  critical  to  the  measurement  of  risk  and 
uncertainty.  These  factors  of  risk  and  uncer¬ 
tainty  as  measured  by  expert  opinion  contri¬ 
bute  to  the  subjective  probability  that  an 
event  will  occur  with  a  specific  probability. 
Several  authors  were  quite  concerned  with  the 
reliability  of  so-called  expert  ’testimony'. 
Several  others  were  very  concerned  about  the 
ability  to  determine  the  relative  accuracy  of 
these  estimates  and  none  were  able  to  suggest 
methods  of  determining  the  adequacy  of  the 
expert  opinion.  In  fact,  H.  M.  Parsons  con¬ 
cluded  that  the  reliance  on  system  designers 
for  the  opinion  and  preferences  of  experts  is 
foolhardy.  Such  experts  may  provide  suggestive 
leads  but  are  not  reliable  guides  as  demon¬ 
strated  by  repeated  disagreement  with  objective 
data.  In  several  technologies,  the  aggregated 
group  opinion  or  concensus  is  used  as  the  value 
to  be  entered  in  the  risk  analysis  model.  It 
is  questionable  whether  cr  not  this  group 
response  can  be  aggregated  in  a  meaningful 
manner.  There  is  no  way  to  evaluate  the  agg¬ 
regated  group  response.  This  creates  some 
serious  difficulties  because  generally  group 
opinions  are  used  as  entering  arguments  in 
risk  analysis  models.  There  is  evidence  that 
external  pressure  to  conform  to  the  popular  or 
top-level  management  preferences  or  a  desire  to 
avoid  rocking  the  boat  may  seriously  affect  the 
group  decision-making  process.  The  impact  of 
the  leader  upon  the  group  was  discussed  by 
several  authors  and  was  generally  agreed  upon 
to  be  one  of  the  most  serious  difficulties  in 
evaluating  the  parameters  for  the  risk  assess¬ 
ment  models.  Another  area  that  impacts  on  the 
accuracy  of  the  subjective  probabilities  is 
the  accountability  of  the  participants.  It  was 
suggested  that  lacking  accountability,  a  par¬ 
ticipant  cannot  blame  nameless  others  for  any 
findings  he  does  not  like.  Several  authors 
conclude  that  valid  techniques  to  collect  sub¬ 
jective  judgments  are  neither  available  nor 
are  they  likely  to  be  developed  in  the  immedi¬ 
ate  future.  Several  also  mentioned  that  at 
the  present  time  a  complete,  well-documented, 
real-life  case  study  of  a  major  development 
program  is  still  very  much  needed  to  bridge 
the  credibility  gap  between  practice  and 
theory  in  the  area  of  risk  analysis. 

There  is  even  difficulty  in  developing  subjec¬ 
tive  judgments  within  the  contractor  organiza¬ 


tions.  The  contractor  may  have  difficulty  in 
getting  his  own  experts  to  accurately  transmit 
their  perceptions  upward  to  their  superiors, 
if  they  perceive  these  values  to  be  'out-of- 
line 1 .  If  it  is  difficult  for  the  contractor 
to  develop  subjective  probabilities,  it 
becomes  impossible  for  the  system  program 
office  who  is  responsible  for  the  overall  per¬ 
formance  to  evaluate  the  validity  of  these  sub¬ 
jective  probabilities.  As  time  passes  in  the 
weapons  acquisition  process,  there  is  indica¬ 
tion  that,  should  the  contractor  performance 
be  below  what  is  expected,  the  contractor  may 
show  great  reluctance  to  provide  unfavorable 
information  to  the  system  program  office, 
especially  in  the  situation  like  the  A-10 
where  a  company's  virtual  survival  as  a  prime 
contractor  may  depend  upon  an  impending  pro¬ 
duction  decision.  Given  that  the  data  from 
the  contractor  is  subject  to  wide  variability 
— depending  upon  the  level  of  expertise,  the 
background  in  decision  analysis,  the  back¬ 
ground  of  the  so-called  experts  in  probability 
theory,  the  management  philosophy  of  the  com¬ 
pany,  and  the  importance  of  the  contract — it 
is  unlikely  that  the  system  program  office 
would  receive  from  the  contractor  highly 
credible  data.  If  unacceptable  data  is 
received  from  the  contractor,  it  is  extremely 
difficult  for  the  system  program  office  to 
develop  acceptable  estimates  for  the  risk 
analysis  programs. 

Presently,  the  decision-making  systems  or  risk 
analysis  programs  are  designed  for  the  user 
and  this  would  include,  of  course,  the  con¬ 
tractor  and  the  system  program  office  personnel 
rather  than  for  the  task  at  hand.  Several 
authors  indicate  that  it  might  be  more  cost- 
effective  to  develop  specific  risk  analysis 
programs  for  specific  programs  based  on  the 
relative  experience  of  the  personnel  at  the 
contractor  office,  the  system  program  office, 
and  at  the  decision-making  level. 

Regardless  of  the  sophistication,  elaboration, 
or  expertise  of  the  scientists  and  engineers 
who  are  providing  the  scientific  data  that  is 
entered  into  the  risk  analysis  models,  there 
is  no  indication  at  the  present  time  that  the 
modeling  techniques  are  capable  of  handling 
irrational  acts  either  by  man  or  by  nature. 

For  instance,  to  have  predicted  that  the 
United  States  would  currently  have  52  persons 
hostage  in  Iran,  a  friendly  country  only  two 
years  ago,  is  a  problem  that  would  probably  not 
have  been  modeled  two  years  ago.  Pending  Con¬ 
gressional  findings,  alterations  that  may  delay 
production  as  much  as  12  months  with  the  asso¬ 
ciated  rise  in  cost  are  difficult  if  not 
impossible  to  model  in  an  objective  manner. 
These  particular  difficulties  have  led  many  to 
believe  that  would  be  foolhardy  to  expect 
a  valid  risk  analysis  for  every  major  program 
and  therefore  at  the  present  time  there  are 
indications  that  in  order  to  comply  with  the 
Department  of  Defense  requirements  many  pro- 
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gram  offices  are  merely  giving  risk  analysis 
formal  lip  service. 

Many  authors  were  concerned  about  having 
experts  assign  particular  values  in  the  risk 
models  when  the  personnel  who  were  assigning 
these  particular  values  had  no  particular 
stake  in  the  results.  Similarly,  they  were 
concerned  with  the  values  that  would  be  uti¬ 
lized  by  those  whose  very  jobs  and/or  careers 
could  be  significantly  affected  by  the  deci¬ 
sion  concerning  a  particular  weapons  system. 

In  the  area  of  weapons  acquisition,  specifi¬ 
cally,  there  was  a  great  amount  of  data  that 
indicated  that  the  experts  were  constantly 
overly  optimistic. 

In  the  area  of  small  group  behavior,  it  is 
interesting  to  note  that  a  decision  made  by  a 
particular  group  at  a  particular  time  on  a 
particular  subject  could  be  altered  at  a  later 
time  on  a  similar  subject.  There  appears  to 
be  a  number  of  outside,  interactive  forces 
that  can  impact  upon  the  decision  arrived  at 
by  a  single  group.  As  far  as  group  decisions, 
it  is  generally  conceded  that  group  decisions 
are  better  than  individual  decisions.  This  is 
based  primarily  on  the  group  dynamics  and  the 
interaction  of  the  participants  within  the 
group.  Additional  data  supports  the  synergism 
of  group  dynamics.  Several  authors  show  how 
to  improve  this  synergism  and  almost  all  agree 
that  the  groups  are  more  risky  than  the  indi¬ 
viduals.  However,  there  are  presently  no  for¬ 
mal  screening  or  evaluation  procedures  or  com¬ 
puter  programs  that  can  evaluate  the  decisions 
made  by  the  individual  groups.  In  developing 
group  solutions,  there  is  a  great  dependence 
upon  the  group  members  asking  the  right  ques¬ 
tions  and  then  being  able  to  answer  these 
questions  in  a  scientific  manner.  Again, 
given  the  fact  that  the  right  questions  are 
asked  and  the  right  answers  are  given,  there 
is  difficulty  in  measuring  the  effectiveness 
of  these  particular  numbers  as  applied  to  the 
risk  analysis  process.  There  are  indications 
that  individual  group  members  may  concern  them¬ 
selves  more  with  reaching  concensus  than  with 
the  quality  of  the  agreed  judgment.  Various 
factors — such  as  individual  dominance  through 
personality  characteristics  or  rank  or  posi¬ 
tion  within  the  organization — may  influence 
the  judgment  of  the  individuals  in  the  group 
and  therefore  the  group  concensus.  This  is 
particularly  of  interest  since  the  individual 
characters  in  the  drama  are  irrelevant  to  the 
task  and/or  the  evaluations  or  numbers  that 
arc  to  be  provided  for  the  risk  analysis.  The 
researchers  investigated  several  behavioral 
interaction  techniques  including  the  Delphi 
method,  the  nominal  group  technique,  the  modi¬ 
fied  nominal  group  technique,  the  concensus 
technique,  and  the  no  interaction  technique. 
Authors  differ  as  to  which  provides  the  best 
group  solutions.  Some  favor  the  Delphi  method 
and  some  the  nominal  group  technique.  In  order 
to  improve  the  group  concensus — and  here  when 


we  speak  of  the  group  concensus,  we  are  talk¬ 
ing  about  a  group  of  experts — it  was  suggested 
that  possibly  the  training  of  these  experts  in 
probabilistic  thinking  could  lead  to  signifi¬ 
cant  improvement  in  their  quantification  of 
uncertainty.  One  author  seemed  to  indicate 
that  interaction  tends  to  increase  the  cer¬ 
tainty  of  the  group,  decreases  the  calibration, 
and  decreases  disagreement  among  group  members. 
However,  in  many  instances,  simple  averaging 
of  individual  assessments  without  any  group 
interaction  may  be  the  most  desirable,  simply 
because  it  is  the  easiest  to  use. 


CONCLUSION 


The  authors  were  able  to  determine  that  there 
was  very  little  information  that  directly 
addressed  the  small  group  behavior  in  uncertain 
decision  environments  such  as  the  acquisition 
process.  There  was  great  disagreement  among 
experts  concerning  the  validity  of  the  differ¬ 
ent  group  dynamics  processes.  All  generally 
agreed  that  better  data  for  risk  analysis 
models  could  be  obtained  if  personnel  could  be 
trained  in  probabilistic  methods  and  would  be 
held  responsible  for  the  decisions  that 
resulted  from  the  output  of  the  group  analysis. 
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INTRODUCTION 

Extensive  research  has  been  accomplished  in 
the  name  of  procurement  research  over  the 
past  few  years  but  no  definitive,  de¬ 
limited  concept  has  evolved  as  to  what 
constitutes  procurement  research  (15).  A 
review  of  the  early  Department  of  Defense 
(DOD)  Procurement  Symposia  "Proceedings" 
indicated  that  professionals  in  the  field 
of  procurement  called  for  a  definition  of 
the  term  " Procurement  Research"  as  well 
as  the  classification  of  its  character¬ 
istics  into  a  model  to  provide  more 
efficient  use  of  resources.  The  need 
existed  to  clearly  define  procurement 
research  and  to  classify  its  character¬ 
istics  into  a  useable  conceptual  model. 
Consequently,  a  study  was  initiated  at 
the  Air  Force  Institute  of  Technology 
to  accomplish  these  goals  (4). 

Research  has  been  viewed  by  many  in  the 
field  as  a  key  to  alleviating  both  exis¬ 
ting  and  future  procurement  problems  (2:1). 
Senator  Stennis,  Chairman  of  the  Senate 
Committee  on  Armed  Services,  and  Congress¬ 
man  Price,  Chairman  of  the  House  Com¬ 
mittee  on  Armed  Services,  reiterated 
this  widely  held  belief  in  a  joint  let¬ 
ter  to  Former  Defense  Secretary  Schles- 
inger: 

"We  recognize  the  value  and  importance  of 
procurement  research  as  a  means  of  improv¬ 
ing  the  procurement  process-one  of  the 
most  crucial  tasks  in  Government  (10)." 

Even  though  there  is  a  general  consensus  as 
to  the  importance  and  possible  impact  of 
research  on  the  procurement  process, 
procurement  research  as  a  discipline  has 
not  been  clearly  defined  in  existing 
literature  and  practice.  Also,  there  is 
little  agreement  among  the  agencies  per¬ 
forming  this  research  as  to  what  consti¬ 
tutes  procurement  research  (1:2).  There 
have  been  several  indications  as  to  a 
possible  scope  for  procurement  research; 
for  example,  J.M.  Malloy,  then  Deputy 
Assistant  Secretary  of  Defense  for  Pro¬ 
curement,  described  procurement  research 
as  essentially  "a  systematic  approach" 
that  follows  the  scientific  method  (8:215). 
However,  there  has  not  been  a  concerted 
effort  to  adequately  describe  what  should 
be  included.  The  result  has  been  that 
research  efforts  are  categorized  as 
both  tentative  and  diffuse  (11:4). 


Robert  Judson,  then  Deputy  Director  of 
Commission  Studies,  Commission  on  Gov¬ 
ernment  Procurement  ( COG P ) ,  Stated  that 
procurement  research's 

.  .first  order  of  priority  .  .  .is  to 
construct  .  .  .a  model  so  that  we  can 
share  a  consensus  on  procurement  problems, 

.  .  .a  comprehensive  studious  critical 
conceptual  model  for  the  acquisition 
process  that  will  give  us  insights  we  do 
not  now  posses  that  will  help  us  identify 
what  we  don't  know  (6:93)." 

The  importance  of  procurement  research  and 
the  necessity  of  defining  its  role  in 
government  acquisition  was  reaffirmed  in 
an  interview  with  Robert  F.  Trimble,  then 
Assistant  Administrator,  Office  of 
Federal  Procurement  Policy  (OFPP)  for  Con¬ 
tract  Administration.  He  discussed  pro¬ 
curement  research  as  follows: 

"I've  long  had  an  interest  in  procurement 
research.  I  think  that  it  (an  attempt  to 
define  and  classify  procurement  research) 
is  one  that  is  particularly  important  be¬ 
cause  I  have  seen  a  considerable  amount 
of  confusion  regarding  what  constitutes 
procurement  research.  I  believe  that 
this  matter  needs  to  be  clarified  so  that 
we  can  more  efficiently  utilize  the  man¬ 
power  resources  that  we  have  in  this 
particular  area  (15)." 

BACKGROUND 

In  the  past,  specific  areas  in  procurement 
research  have  not  been  clearly  delimited, 
thus,  a  historical  background  must  con¬ 
centrate  on  the  procurement  organizations 
which  have  developed  during  the  past 
twenty-five  years.  The  evolution  of 
procurement  research  has  been  character¬ 
ized  by  changes  in  organization  and  pro¬ 
cedures.  Research,  per  se,  has  not  been 
emphasized;  rather,  the  emphasis  has  been 
on  the  changes  in  DOD  and  Air  Force 
procurement  organization  which  resulted 
from  the  need  for  better  procurement 
methods.  The  lack  of  a  clear  definition 
of  just  what  procurement  research  includes 
made  this  approach  necessary. 

In  the  1950's,  various  attempts  were  made 
to  have  money  through  reorganizing  and 
central i zaing  purchases  of  common  items. 
The  mertia  of  old  techniques  was  slow  to 
yield  to  change;  moreover  each  military 
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service  was  "isolated"  from  the  others  as 
far  as  procurement  methods.  In  the  1960's, 
some  efforts  were  made  to  exchange  pro¬ 
curement  information  and  to  evaluate 
decision-making  during  the  acquisition  pro¬ 
cess.  New  approaches  were  being  utilized 
to  improve  the  management  information  flow. 
In  the  1970 *  s ,  many  changes  occured  in  the 
formal  acquisition  process.  After  many 
long  years  of  inefficiency  and  redundancy, 
the  national  procurement  policy,  education, 
and  research  are  becoming  centralized  and 
coordinated  under  the  Federal  Acquisition 
Institute  (FAI). 

Mr.  Robert  Judson,  then  Deputy  Director, 
Commission  Studies,  Commission  on  Govern¬ 
ment  Procurement  (COGP),  in  an  address  to 
the  second  DOD  Procurement  Symposium  in 
1973  made  a  challenge  to  the  procurement 
profession.  He  said: 

"You,  gentlemen,  have  a  golden  opportunity 
to  redirect  procurement  research  to  achieve 
new  goals  of  excellence.  First  let’s  do 
our  research  on  the  problems  of  research 
before  we  lose  the  chance  to  make  pro¬ 
curement  what  we  want  it  to  be  (6.99)" 

This  challenge  was  re-emphasized  by  Dr. 

John  J.  Bennett,  then  Acting  Assistant 
Secretary  of  Defense  (Installations  and 
Logistics)  in  the  Defense  Management 
Journal ,  Jul y  1975 : 

"Procurement  research  is  not  yet  a  house¬ 
hold  phrase  in  the  Department  of  Defense... 
It  needs  a  great  deal  of  attention  from 
management  and  those  people  actually  en¬ 
gaged  in  procurement  projects  (2:1)" 

In  summary,  the  important  .events  in  the 
evolution  of  procurement  research  start 
with  the  Second  Hoover  Commission  in  1953 
and  continue  up  to  the  present  time.  Key 
events  in  procurement  reorganization  in  the 
I960'  s  include  the  reorgani  zation  of  Air 
Force  Systems  Command  and  Air  Force  Logis¬ 
tics  Command,  the  Hershey  Procurement 
Pricing  Conference,  the  establishment  of 
the  Army  Procurement  Research  Office,  and 
the  Commission  on  Government  Procurement. 
The  significant  events  (thus  far)  during 
the  1970' s  include  the  six  DOD  Procurement 
Symposia,  the  establishment  of  the  Air 
Force  Business  Research  Management  Center, 
the  addition  of  a  Graduate  Procurement  cur¬ 
riculum  to  the  Air  Force  Institute  of 
Technology,  School  of  Systems  and  Logis¬ 
tics,  and  a  Systems  Acquisition  Management 
curriculum  to  the  Naval  Postgraduate 
School  ,  the  establishment  of  the  Office 
of  Federal  Procurement  Policy,  and  the 
founding  of  the  Federal  Acquisition  In- 
sti tute. 


BASIC  STUDY  OBJECTIVES 

The  following  research  objectives  of  the 
study  are  germane: 

1.  To  define  procurement  research  so  that 
a  common  foundation  can  be  used  when 
discussing  this  subject. 

2.  To  classify  procurement  research  efforts 
and  functions  into  various  areas  and  to 
identify  those  areas  that  are  most  fre¬ 
quently  investigated. 

3.  From  these  classifications,  to  suggest 
a  detailed  al  gori  thm1  which  can  be  used  for 
deciding  if  an  effort  is  procurement 
research. 

METHODOLOGY 

A  literature  review  was  initiated.  It 
disclosed  an  increasing  interest  in  the 
area  of  procurement  research  and  in  de¬ 
fining  procurement  research,  but  no 
suggestions  were  made  as  to  how  this 
specific  area  of  research  should  be  defined 
or  how  it  should  be  classified  from  a 
taxonomical  standpoint.  A  search  dis¬ 
closed  that  content  analysis  provided  a 
rigorously  subjective  technique  for  group¬ 
ing  various  procurement  efforts:  by  divi¬ 
sion  of  scientific  study,  by  breadth  of 
application,  by  degree  of  control  ,  by 
level  of  outcome,  by  level  of  effort,  and 
by  placement  in  the  acquisition  and 
procurement  processes.2  Through  a  system 
of  summarizing  and  categorizing,  these 
various  groups  were  used  to  suggest  a 
definition  for  procurement  research. 

The  basic  research  design  was  divided  into 
five  areas: 

1.  Classifying  procurement  research  efforts 
and  functions  into  categories  and  sub¬ 
categories  . 

2.  Identifying  the  areas  of  procurement 
research  that  were  mostly  frequently 
investigated. 

3.  Defining  procurement  research  in  terms 
of  characteristics  which  were  evidenced 

in  the  study. 

4.  Suggesting  a  taxonomy  of  procurement 
research. 

5.  Designing  a  procurement  research 
algorithm  for  evaluating  research. 

The  first  three  areas  of  the  research 
design  were  planned  to  answer  the  first 
and  second  research  objectives.  The 
fourth  and  fifth  design  areas  were  planned 
to  answer  the  second  and  third  objectives. 

The  first  design  area  was  planned  to  iden¬ 
tify  specific  scientific  and  research 
characteristics  of  procurement  research  as 
evidenced  in  the  "Proceedings".  Through 
content  analysis,  the  articles  of  the 
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The  results  of  the  content  analysis  are 
limited  to  the  actual  sub-population  itself, 
but  the  sub-population  of  the  "Proceedings" 
represents  an  important  cross-section  of 
recent  DOD  procurement  research  experience. 
Information  derived  from  the  analysis  of 
this  sub-population  can  suggest  important 
characteristics  and  relationships  of 
other  procurement  research  efforts. 

The  areas  of  emphasis  in  the  procurement 
research  of  the  "Proceedings"  were  iden¬ 
tified  in  the  content  analysis.  Procurement 
research  was  characterized  as  a  social 
science  with  abstract  science  combined 
more  often  than  not.  Its  structure  is 
delimited  in  Figure  1.  Efforts  were 
primarily  applied  to  solving  problems. 

The  research  was  primarily  accomplished 
through  a  selected  aggregation  of  information 
(library)  and  the  level  of  outcome  was  des¬ 
criptive.  In  the  sub-population,  the  level  of 
effort  was  primarily  a  professional  paper/re¬ 
search  monograph.  The  relationship  of  procure¬ 
ment  research  with  the  acquisition  process  or 
were  not  concerned  with  acquisition  process  at 
all.  Emphasis  in  the  procurement  process  was 
primarily  in  the  pre-award  phase  with  many 
articles  dealing  with  more  than  one  phase. 

The  definition  of  procurement  research  and  the 
classification  of  the  characteri sties  of  pro¬ 
curement  research  were  combined  with  the  in¬ 
formation  from  the  literature  review  to  de¬ 
velop  the  taxonomy.  The  emphasis  for  the  tax¬ 
onomy  has  been  to  cover  all  possible  areas  of 
procurement  research  as  suggested  by  various 
information  sources.  (See  page  82) 

The  procurement  process  is  the  foundation  upon 
which  procurement  research  is  based  (see  fig¬ 
ure  2).  Procurement  research  can  involve  both 
the  procurement  and  acquisition  processes  and 
their  interrelationship.  Therefore,  to  con¬ 
struct  a  taxonomy  of  procurement  research  fo¬ 
cusing  primarily  on  the  procurement  process,  it 
was  necessary  to  build  a  model  of  this  process. 
Since  procurement  research-is  concerned  with 
the  procurement  process  and  the  procurement 
process  as  an  integral  part  of  the  acquisition 
process,  a  taxonomy  of  the  procurement  pro¬ 
cess  can  serve  as  a  foundation  for  a  taxonomy 
of  procurement  research.  The  areas  and  issues 
pertaining  to  these  processes,  therefore,  also 
'pertain  to  procurement  research.  These  areas 
and  issues,  as  related  to  the  procurement  pro¬ 
cess  were  the  focal  point  of  this  research  and 
the  descriptors  of  the  research  taxonomy. 
Content  analysis  provided  the  general  charac¬ 
teristics  of  procurement  research  and  a  par¬ 
tial  structure  of  the  procurement  processes. 
However,  to  complete  the  taxonomy  of  the  pro¬ 
cess,  it  was  necessary  to  conduct  interviews 
and  make  literature  reviews. 


The  taxonomy  was  constructed  to  display 
five  levels  of  the  procurement  process. 

The  first  level  is  the  procurement 
process.  The  second  level  is  the  three 
phases  (Pre-Award,  Award,  and  Post- 
Award).  The  third  level  is  comprised  of 
the  cycles  that  make  up  each  of  the 
phases.  The  fourth  level  is  a  continuum 
of  events  (Procurement  Continuum)  that 
describes  the  necessary  actions  pertinent 
to  the  life  of  a  "Procurement".  The  fifth 
level,  the  lowest  level  presented,  is 
composed  of  a  number  of  issues  related  to 
each  of  the  events.  (See  Figure  3  page  83) 


The  research  effort  was  adjusted  as  the 
researchers  discovered  new  information 
that  impacted  the  definition  and  the  tax¬ 
onomy.  The  taxonomy  developed  by  the 
researchers  is  outlined  in  Figure  4. 
Lastly,  an  algorithm  was  developed  (See 
Figure  5)  which  followed  the  format  of  a 
decision  flow  chart  with  eight  decision 
points,  all  of  which  (except  one)  must 
be  answered  with  an  affirmative  response 
before  an  effort  can  be  considered 
procurement  research.  These  eight 
decision  points  are  as  follows: 

1.  Is  the  effort  concerned  with  satis¬ 
fying  a  perceived  DOD  need?  Is  the  effort 
attempting  to  solve  a  problem,  provide 
insight  into  an  issue,  or  describe  a 

(See  pages  84  -  8  1 


problem  with  the  DOD?  If  it  is  not,  it 
should  be  considered  for  further  research. 

2.  Is  the  research  effort  concerned  with 
the  acquisition  process?  If  the  answer 
to  this  is  negative,  a  second  question 

is  asked,  "Is  the  research  concerned 
with  the  procurement  process?"  (See 
Figure  4  and  Table  3.)  If  the  answer  to 
this  is  positive,  the  effort  is  retained 
for  further  analysis. 

3.  Is  the  research  concerned  with  the 
procurement  process?  Here,  the  efforts 
judged  affirmatively,  using  question  2, 
are  analyzed  according  to  the  procurement 
taxonomy  and  questions  set  forth  in 

Tab!  e  3. 

4.  Does  the  effort  suggest  a  method  for 
improving  the  knowledge  associated  with 
the  procurement  process?  Three  general 
questions  can  be  asked  of  the  research 
effort. 
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FIGURE  1.  THE  DELIMITATION  OF  PROCUREMENT  RESEARCH 
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FIGURE  2.  THE  PROCUREMENT  PROCESS  (PHASES)  (9) 
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FIGURE  4  (cont'd)  A  PROCUREMENT  TAXONOMY 
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FIGURE  4  (CONT'D)  A  PROCUREMENT  TAXONOMY 
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FIGURE  5,  PROCUREMENT  RESEARCH  ALGORITHM 
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TABLE  3 

CRITERIA  FOR  DETERMINING  WHETHER  A  RESEARCH 
EFFORT  IS  PROCUREMENT  RELATED 

1.  IS  IT  CONCERNED  WITH  ISSUES  PERVASIVE  TO  THE  PROCUREMENT  PROCESS 
(PROCUREMENT  EHICS,  CONTRACT  MANAGEMENT,  TRAINING,  ORGANIZATION)? 

2.  IS  IT  RELATED  TO  REQUIREMENTS  DETERMINATION? 

3.  IS  IT  CONCERNED  WITH  REQUIREMENTS  SPECIFICATION? 

4.  IS  IT  CONCERNED  WITH  PR  ISSUANCE? 

5.  IS  IT  CONCERNED  WITH  OTHER  ASPECTS  OF  THE  REQUIREMENTS  CYCLE? 

6.  IS  IT  RELATED  TO  THE  RECEIPT  OF  A  PR? 

7.  IS  IT  CONCERNED  WITH  PRE-SOCICITATI ON  REVIEW? 

8.  IS  IT  CONCERNED  WITH  FORMAL  SOLICITATION  ISSUANCE? 

9.  IS  IT  CONCERNED  WITH  OTHER  ASPECTS  OF  THE  PR/MIPR  CYCLE? 

ID.  IS  IT  CONCERNED  WITH  THE  RECEIPT  OF  RESPONSES  TO  SOLICITATION? 

11.  IS  IT  RELATED  TO  THE  TECHNICAL  EVALUATION  OF  RESPONSES? 

12.  IS  IT  CONCERNED  WTTH  PROPOSAL  AUDITS? 

13.  IS  IT  CONCERNED  WITH  COST  ANALYSIS/PRICE  ANALYSIS? 

14.  IS  IT  CONCERNED  WITH  PRE-NEGOTIATION  ISSUES? 

15.  IS  IT  CONCERNED  WITH  CONTRACT  NEGOTIATION? 

16.  IS  IT  CONCERNED  WITH  CONTRACTOR  SELECTION? 

17.  IS  IT  CONCERNED  WITH  OTHER  ASPECTS  OF  THE  SOLICITATION/EVALUATION  CYCLE? 

18.  IS  IT  CONCERNED  WITH  SOME  ASPECT  OF  THE  PRE-AWARD  PHASE? 

19.  IS  IT  CONCERNED  WITH  CONTRACT  FUNDING? 

20.  IS  IT  CONCERNED  WITH  CONTRACT  WRITING? 

21.  IS  IT  CONCERNED  WITH  THE  REVIEW  OF  A  CONTRACT  PRIOR  TO  ANNOUNCEMENT 
AND  FINAL  SIGNATURE? 

22.  IS  IT  CONCERNED  WITH  AWARD  ANNOUNCEMENT  PROCEDURES? 

23.  IS  IT  CONCERNED  WITH  CONTRACT  DISTRIBUTION  PROCEDURES? 
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24.  IS  IT  CONCERNED  WTTH  OTHER  ISSUES  OF  THE  AWARD  CYCLE  OR  AWARD  PHASE? 

25.  IS  IT  CONCERNED  WITH  THE  ASSIGNMENT  OF  THE  CONTRACT  FOR  ADMINISTRATION? 

26.  IS  IT  CONCERNED  WITH  CONTRACT  SYSTEM  COMPLIANCE? 

27.  IS  IT  CONCERNED  WITH  PERFORMANCE  MEASUREMENT  OF  THE  CONTRACTOR? 

28.  IS  IT  CONCERNED  WITH  QUALITY  ASSURANCE/PRODUCT  ACCEPTANCE? 

29.  IS  IT  CONCERNED  WITH  CONTRACT  MODIFICATIONS? 

30.  IS  IT  CONCERNED  WITH  PRODUCT  DELIVERY? 

31.  IS  IT  CONCERNED  WITH  CONTRACTOR  PAYMENT? 

32.  IS  IT  CONCERNED  WITH  CONTRACT  WARRANTIES? 

33.  IS  IT  CONCERNED  WITH  CONTRACT  DISPUTES? 

34.  IS  IT  CONCERNED  WITH  REMEDIES  RESULTING  FROM  CONTRACT  DISPUTES? 

35.  IS  IT  CONCERNED  WITH  CONTRACT  COMPLETION  AND  CLOSE-OUT? 

36.  IS  IT  CONCERNED  WITH  ANY  OTHER  ISSUES  OF  THE  CONTRACT  ADMINISTRATION 

CYCLE  OR  THE  POST-AWARD  PHASE? 

37.  IS  IT  CONCERNED  WITH  POST  CONTRACT  ISSUES  SUCH  AS  RENEGOTIATION? 

IF  THE  ANSWER  TO  ANY  OF  THE  ABOVE  QUESTIONS  IS  "YES",  THEN  THE  EFFORT  IS 
RELATED  TO  PROCUREMENT. 


of  the  symposia  "Proceedings"  have  been 
classified  into  various  categories  and 
sub-categories  of  characteristics.  These 
scientific  and  research  characteristics 
have  been  correlated  with  areas  of  the 
procurement  and  acquisition  process. 

To  satisfy  the  first  and  second  objec¬ 
tives,  the  methodological  approach  of 
semantic  content  analysis  was  adopted. 

From  the  universe  of  research,  the  popu¬ 
lation  called  procurement  research  was 
chosen.  This  population  was  further 
narrowed  to  the  sub-population  of 
procurement  research  as  reported  in  the 
"Proceedings"  of  the  five  OOD  Procure¬ 
ment  Research  Symposia.3  The  analysis 
consisted  of  a  census  of  the  total  sub¬ 
population  of  articles  in  these  "Pro¬ 
ceedings". 

To  validate  the  coding,  a  pilot  study 
was  accomplished.  To  enhance  the  re¬ 
liability  of  the  research  effort,  a 
"target"  reliability  percentage  of  902 
was  achieved  during  the  pilot  study. 
Additionally,  during  the  analysis,  random 
samples  of  articles  coded  by  one  resear¬ 
cher  were  recoded  by  a  second  researcher 
to  insure  consistent  and  standard  results. 

After  coding  the  data  for  each  "Pro¬ 
ceedings",  a  relative  frequency  count  of 
occurrences  under  each  digit  code  was 
tabulated.  Each  digit  in  the  seven¬ 
digit  code  represented  a  category  of 
science,  research,  the  acquisition 
process,  or  the  procurement  process.  The 
first  digit  was  coded  to  show  the  division 
of  science  used  in  the  research.  The 
second  digit  was  coded  to  show  the  breadth 
of  application  of  the  research  tech¬ 
niques  used.  The  third  digit  was  coded 
to  identify  the  amount  of  control  used 
by  the  researcher  and  where  the  research 
was  conducted.  The  fourth  digit  was 
coded  to  determine  the  level  of  outcome 
of  the.  research  effort,  what  could  be 
said  about  the  area  studied,  did  it 
describe  a  situation,  or  could  a  model 
be  developed  to  predict  future  events? 

The  fifth  digit  was  coded  to  indicate 
the  level  of  effort  used  in  the  research, 
i.e.,  the  amount  of  time  and  depth  of 
effort  necessary  to  accomplish  the  re¬ 
search.  The  sixth  and  seventh  digits 
were  coded  to  indicate  the  phases  of 
the  acquisition  and  procurement  proces¬ 
ses  with  which  the  research  was  concerned. 

In  the  second  design  area,  the  results  of 
the  content  analysis  were  combined  into 
relative  frequency  distributions.  Each 
sub-category  was  analyzed  to  determine 
those  areas  of  procurement  research  which 
were  most  frequently  investigated  and  which 


were  most  prevalent  in  the  population. 

The  third  research  design  area,  defining 
procurement  research  in  terms  of  charac¬ 
teristics  evidenced  in  the  study,  was 
addressed  using  the  tabulated  data.  The 
characteristics  of  research  and  science 
derived  from  content  analysis  were  com¬ 
bined  with  information  obtained  from 
literature  reviews  and  personal  inter¬ 
views  to  develop  a  tentative  conceptual 
definition  of  procurement  research. 

The  fourth  area  of  research  design  was 
planned  to  classify  procurement  research 
efforts  into  various  areas,  as  stated  in 
the  second  objective,  and  to  suggest  a 
detailed  procurement  taxonomy.  Infor¬ 
mation  for  this  area  was  gathered  from 
existing  literature  and  personal  inter¬ 
views  (7,  9,  12,  and  13) . 

In  the  fifth  research  design  area,  the 
designing  of  a  detailed  algorithm  was 
envisioned  to  meet  the  third  objective. 
This  algorithm  could  be  employed  in  de¬ 
termining  whether  an  effort  within  000 
is  related  to  procurement  and  whether  it 
is  research.  The  algorithm  was  derived 
from  information  gained  from  the  content 
analysis,  literature  reviews  and  personal 
interviews  as  noted  in  the  aforementioned 
paragraph. 

FINDINGS 

Data  from  the  content  analysis  of  the 
"Proceedings"  have  been  tabulated,  and 
a  taxonomy  and  algorithm  have  been 
developed.  The  results  of  the  content 
analysis,  which  are  listed  in  Table  1, 
showed  the  following  primary  areas  of 
emphasis  as  related  to  the  selected 
criteria. 

(See  page  g]  ) 


CONCLUSIONS 

The  results  of  the  research  suggested  the 
following  definition  of  procurement 
research: 

Procurement  research  (and  acquisition 
research)  is  an  applied  science  using 
the  characteristics  of  the  social 
sciences  in  combination  with  mathematical 
sciences  to  solve  procurement  problems. 

It  tends  to  rely  heavily  on  the  use  of 
previously  gathered  data  to  seek  solutions 
to  problems,  equally  dividing  its  efforts 
between  the  acquisition  process,  emphasis 
is  on  the  pre-award  phase  in  an  effort 
to  identify  cost-related  problems. 
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national  contract  MANAGEMENT  QUARTERLY  JOURNAL  39 
TABLE  1  -  SUMMARY  OF  FINDINGS 


1. 

2. 

3. 

4. 

5. 

6. 

7. 


DIVISION  OF  SCIENCE:  SOCIAL  (46%) 

ABSTRACT/SOCIAL  COMBINED  (47%) 

BREADTH  OF  APPLICATION:  APPLIED  (62%) 

DEGREE  OF  CONTROL:  LIBRARY  (54%) 

LEVEL  OF  OUTCOME:  DESCRIPTIVE  (55%) 


LtVEL  OF  EFFORT: 


PROFESSIONAL  PAPER/RESEARCH  MONOGRAPH  (68%) 


PHASE  OF  THE  ACQUISITION  PROCESS- 
MORE  THAN  ONE  PHASE 

NOT  CONCERNED  WITH  THE  ACQUISITION  PROCESS  (47%) 


PHASE  OF  THE  PROCUREMENT  PROCESS 
PRE-AWARD  (47%) 

MORE  THAN  ONE  PHASE  (33%) 


•  rtkles,  th,  s»ur«  U  rSrSd  (“4) 


TABLE  2  -  ARTICLE  SOURCE  DISTRIBUTION 


AGENCY 

NUMBER 

PERCENTAGE 

DOD 

y 

8% 

ARMY 

19 

17% 

NAVY 

14 

AIR  FORCE 

12% 

39 

34  & 

NON  DOD 

Federal 

Agenci es 

11 

10% 

Private 

Business/ 

Uni  vers i ties 

22 

19% 

ine  initial  taxonomy  was  divided  into  five  level  c 
research.  Each  level  subdivided  the  previous  level  l 
areas  where  procurement  research  can  be  Identified 


procurement 
more  specific 
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a.  Does  it  increase  the  uncertainty  of  the 
procurement  process? 

b.  Will  the  suggested  results  of  the 
research  effort  provide  more  knowledge  of 
the  procurement  process,  knowledge  of  an 
area  that  has  not  been  investigated  before? 

c.  Does  the  effort  clarify  events,  areas, 
or  issues  in  the  process? 

5.  Does  the  effort  address  an  identified 
requirement?  Before  passing  judgement  on 
the  effort,  a  review  by  relevant  procurement 
personnel  should  be  obtained  to  establish 
the  validity  of  the  undertaking.  If  judged 
negatively,  the  effort  should  be  discarded 
or  held  for  later  evaluation.  If  judged 
affirmatively,  the  effort  passes  on  to  the 
next  algorithm  question. 

6.  Does  the  effort  suggest  a  solution  or 
make  observations  that  can  ultimately 
improve  the  procurement  process?  An  effort 
that  is  classified  as  procurement  research  r 
should  be  directed  toward  the  improvement 

of  the  procurement  process. 

The  key  to  this  decision  point  in  the 
algorithm  is  that  the  research  effort  con¬ 
tributes  to  the  improvement  of  the  procure¬ 
ment  process  either  through  a  solution  to  an 
existing  problem  or  through  observations 
that  may  lead  to  the  understanding  and 
solutions  to  future  problems. 

7.  Does  the  effort  suggest  a  practical 
plan  for  implementation  or  utilization  of 
the  perceived  results?  An  effort  may  be 
directed  at  the  ultimate  improvement  of 
the  procurement  process.  It  may  suggest  a 
solution  to  an  existing  problem  or  it  may 
be  used  as  a  "stepping  stone"  to  solutions 
of  future  problems.  However,  if  the  effort 
suggests  using  implementation/utilization 
procedures  and/or  techniques  that  woul-d 

be  impractical,  the  effort  is  suspect. 

Application  of  this  decision  point  requires 
a  word  of  caution.  Implementation/utili¬ 
zation  plans  suggested  by  some  research 
efforts  may  be  deemed  impractical  now, 
only  to  be  proven  practical  at  some  future 
point  in  time.  This  fact  may  require  that 
the  final  determination  of  whether  some 
research  efforts  are  procurement  research 
needs  to  be  deferred  to  a  later  date. 

8.  Does  the  effort  have  characteristics  of 
the  scientific  method?  Determination  of 
whether  the  effort  follows  the  scientific 
method  is  made  by  either  the  research  analyst 
and/or  the  procurement  manager  subjecting 

the  effort  to  the  series  of  questions 
listed  in  Table  4.  (If  the  effort  is  pro¬ 
posed,  questions  (1-10)  would  be  applicable 


but  if  the  effort  was  completed  research, 
all  questions  would  be  applicable.)  If 
negative  answers  were  obtained,  the 
effort  should  be  discarded,  or  returned 
to  the  researcher  for  reclarification 
and  rework  and  then  returned  for  further 
analysis  by  the  research  approving 
agency. 

This  algorithm  suggests  that  a  certain 
level  of  effort  be  undertaken  by  the 
researchers  who  proposes  the  study 
prior  to  its  submission  to  the  approving 
agency  for  acceptance.  Resources  avail¬ 
able  to  the  procurement  research  community 
are  necessarily  limited  and  need  to  be 
applied  only  to  pertinent  research 
proposals.  Time  and  funds  cannot  be  ill- 
spent  on  poorly  defined  research  proposals 
that  return  marginal  results  or  have  no 
applicability  to  the  procurement  process. 

WHAT  PROCUREMENT  RESEARCH  SHOULD  BE  .  . 

The  previously  discussed  research  efforts 
concentrated  on  what  procurement  research 
is  and  has  been  during  the  past  few 
years.  From  exposure  to  the  information 
that  was  reviewed  in  this  effort,  the 
research  team  gained  an  insight  into 
procurement  research  and  herein  suggest 
what  procurement  research  should  be: 

1.  It  should  concern  the  acquisition  or 
procurement  processes.  Research  accom¬ 
plished  by  procurement  researchers  that 

(See  page  93) 

does  not  involve  the  procurement  or 
acquisition  processes  makes  an  inefficient 
use  of  limited  resources. 

2.  Procurement  research  should  seek 
solutions  to  procurement  problems.  Pro¬ 
curement  research  should  be  applied 
research;  it  should  be  concerned  with 
seeking  solutions  to  problems  faced  by 
procurement  managers  and  personnel  . 

3.  Procurement  research  should  be  cost 
effective.  Procurement  researchers 
should  concern  themselves  with  a  cost 
analysis  of  their  own  work.  If  the 
research  can  be  performed  at  a  lower  cost 
external  to  the  originating  research 
agency,  then  the  effort  should  be  accom- 
pl ished  externally. 

4.  Procurement  research  should  follow  the 
scientific  method.  The  "Proceedings" 
indicated  that  procurement  research  did 
follow  the  scientific  method  in  its 
approach  to  problem  solving.  Future 
research  should  use  the  same  procedures/ 
techniques . 
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SCIENTIFIC  METHOD  QUESTIONS  LIST 


1.  Does  it  define  the  problem? 

2.  Does  the  effort  survey  existing  pertinent  literature? 

3.  Has  the  researcher  evaluated  past  studies  for  applicability  to  his 
effort? 

4.  Does  the  effort  build  on  previously  developed  knowledge? 

5.  Is  the  scope  defined  and  specified?  And  are  the  specified  objectives 
to  be  met  listed? 

6.  Does  the  effort  suggest  the  testing  of  a  hypothesis  or  the  answering 
of  a  research  question? 

7.  Is  there  a  specified  plan? 

8.  Does  the  effort  list  assumptions/limitations? 

9.  Is  the  methodology  logical  and  appropriate  for  the  objectives  specified? 

10.  Does  the  effort  gather  data  and/or  facts? 

11.  Are  the  data  valid  and  reliable? 

12.  Does  the  effort  report  describe  predict  or  explain? 

13.  Do  conclusions  logically  flow  from  the  data? 

14.  Can  the  effort  be  replicated  to  achieve  consistent  results? 


4 
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5.  Procurement  research  should  be  unbiased. 
Procurement  research  should  report  true 
findings,  not  "channel"  results  to  suit 

the  researchers.  The  researchers  should 
apply  rigorous  subjectivity  to  his  re¬ 
search  and  remain  unbiased  in  his  analysis. 

6.  Procurement  research  should  make  use 
of  the  best  analytical  methods.  Poor 
research  techniques  waste  resources  and 
provide  weak  solutions  to  problems 

that  may  require  strong  remedies.  A 
careful,  thorough  evaluation  of  a  pro¬ 
curement  research  problem  can  suggest 
the  best  analytical  method  to  use  in 
the  effort. 

7.  Procurement  research  should  be 
original  and  not  redundant.  Prior  to 
doing  research,  procurement  people  should 
review  previous  studies  and  ascertain 
whether  a  new  research  effort  is  justi¬ 
fied  or  whether  the  findings  of  a 
previous  study  are  sufficient. 

8.  Procurement  research  should  be  shared. 
Central  procurement  information  storage 
facilities  should  be  accessible  to  all 
procurement  research  organizations,  In¬ 
ternal  and  external  to  the  Government. 
Results  should  be  publicized,  such  as 
those  found  in  the  "Proceedings".  Only 
through  the  sharing  of  information  can 
the  redundancy  be  reduced  and  resources 
saved. 

9.  Procurement  research  should  be 
simple,  yet  accomplish  the  task.  Pro> 
curement  research  should  accomplish  its 
specific  task  in  the  most  direct  method 
possible.  It  should  not  confuse  the 
problem-solving  methodology  with  tech¬ 
niques  designed  to  impress  the  requester 
while  hiding  the  path  the  researcher 
used  to  seek  his  solution.  The  approach 
used  by  the  researcher  should  be  "fair 
and  reasonable"  to  all  parties. 

COROLLARY  OBSERVATIONS  ON 
PROCUREMENT  RESEARCH 

Experience  gamed  during  the  course  of 
this  research  may  prove  enlightening 
to  subsequent  researchers.  These  obser¬ 
vations  are  summarized  as  follows: 

1.  An  increasing  level  of  interest  in 
procurement  was  noted  as  a  definite  trend 
during  the  past  few  years. 

2.  The  annual  DOD  Procurement  Research 
Symposium  offers  an  excellent  means  for 
sharing  procurement  research  information; 
however,  often  other  research  efforts 
and  results  are  not  shared. 


3.  Procurement  researchers  generally  do 
not  share  their  current  progress  or 
projects . 

4.  Often  the  method  or  technique  that 
resulted  from  the  research  effort  could 
be  applied  to  other  problem  situations, 
but  the  research  itself  was  done  strict¬ 
ly  in  response  to  one  problem.  General 
research  to  improve  the  overall  acqui¬ 
sition  and  procurement  processes  was 

1 acking . 

5.  The  present  information  retrieval 
systems  do  not  provide  a  totally  acces¬ 
sible  system  to  the  researcher. 

These  corollary  observations  would  be 
incomplete  without  some  suggestions  or 
recommendations  for  improvement.  Indi¬ 
cation  of  a  deficient  area  Implies  that 
better  methods  are  perceived  for  getting 
the  task  done;  the  next  section  offers 
recommendations  for  improving  procure¬ 
ment  research, 

RECOMMENDATIONS 

Further  studies  must  be  made  and  current 
methods  must  be  changed  in  order  for 
procurement  research  to  be  improved.  The 
results  and  conclusions  from  this  study 
suggest  starting  for  further  studies  and 
alternatives  courses  of  action  for  current 
methods  In  procurement  research.  Eight 
recommendations  for  further  study  and  pro¬ 
curement  research  Improvement  follow: 

1.  Add  research  studies  with  a  longer  range 
perspective  to  present  probl  em/ response  type 
studies.  The  addition  of  some  longer  range 
research  in  procurement  may  identify  influen¬ 
tial  factors  that  are  not  evident  in  the 
short-range,  reactive  approach. 

2.  Areas  of  procurement  research  effort, 
significant  research  progress,  and  research 
results  should  be  shared  with  the  procure¬ 
ment  community.  The  area  of  sharing  infor¬ 
mation  on  procurement  research  is  essential 
to  the  DOD  procurement  community.  Further 
research  should  be  done:.  (1)  to  research 
the  extent  of  the  problem  of  how  many  com¬ 
pleted  procurement  research  studies  do  not 
get  into  the  DOD  information  retrieval 
systems  and  to  correct  this  deficiency  in 
information  flow,  and  (2)  to  find  a  means 
to  identify  current  DOD  procurement  efforts 
in  progress  and  to  publicize  this  information 
on  a  regular  basis. 

3.  The  DOD  should  adopt  the  taxonomy  deve¬ 
loped  In  this  research  effort  as  a  common 
taxonomy  of  procurement  reserach  for  use  by 
its  agencies.  A  standardized  taxonomy  of 
procurement  research  would  allow  researchers 
from  all  DOD  agencies  to  establish  a  common 
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framework  for  communication.  Not  only  would 
researchers  be  on  a  common  base,  but  procure¬ 
ment  people  could  understand  research  results 
from  other  agencies  and  possible  apply  new 
and  better  techniques  to  their  own  work.  The 
procurement  research  taxonomy  presented  in 
this  study  could  be  a  logical  starting  place 
from  which  a  common  DOD  taxonomy  could  be 
expanded.  Finally,  a  common  taxonomy  could 
be  used  as  a  data  base  for  assigning  descrip¬ 
tors  to  procurement  research  in  a  computerized 
information  system. 

4.  The  procurement  research  taxonomy  that  Is 
suggested  in  this  study  should  be  critically 
analyzed  and  expanded.  The  taxonomy  Is  an 
attempt  at  categorizing  the  procurement 
process  and  the  field  of  procurement  research. 
Through  further  study,  this  taxonomy  could  be 
validated  and  expanded  to  include  requirement 
definition  and  use  as  they  impact  upon  pro¬ 
curement. 

5.  Further  algorithms  for  conducting  pro¬ 
curement  research  and  for  deciding  whether  to 
research  procurement  problems  should  be  con¬ 
structed  and  used.  During  the  construction 
of  the  algorithm,  the  researchers  had  much 
difficulty  in  establishing  a  perspective  from 
which  to  construct  the  algorithm.  Procure¬ 
ment  research  can  be  viewed  in  terms  of  a 
given  output  or  ongoing  process,  a  method  for 
conducting  the  research,  or  a:  method  for  de¬ 
ciding  whether  a  porcurement  problem  should 
be  researched  by  an  organization/i ndivldual . 
Future  efforts  should  research  these  two 
areas  to  provide  guidelines  for  conducting 
the  procurement  research  process  and  for 
making  the  important  decision  of  whether  or 
not  to  undertake  a  research  effort. 

6.  A  model  for  DLSIE  (Defense  Logistics 
Systems  Information  Exchange)  abstracts  should 
be  developed  so  that  key  words  would  provide 
ready  relevant  information.  Research  should 
be  done  to  develop  a  model  for  writing  DLSIE 
abstracts  so  that  a  content  anlysls  of  an 
abstract  would  determine  key  words,  words 
similar  to  the  phases,  cycles,  events,  and 
issues  of  the  taxonomy  presented  in  Figure  4. 
Procurement  researchers  could  then  identify 
those  studies  relevant  to  their  areas  of 
specific  research  from  this  content  analysis 
of  the  abstracts. 

7.  A  sequential  analysis  of  procurement 
research  efforts  should  be  performed.  Hood 
and  Strayer  (5)  suggested  that  procurement 
research,  as  a  developing  discipline,  can  be 
portrayed  as  transitioning  six  development 
phases  in  a  sequential  evolutionary  process 
from  a  new  discipline  to  full  maturation. 

The  significance  of  this  evolutionary  process 
and  its  developmental  phases  is  that  each 
phase  differs  in  terms  of  the  kinds  of  ques¬ 
tions  or  issues  addressed  and  types  of 
research  activity  conducted  within  each  phase. 


8.  Research  should  be  done  to  prioritize 
those  "issues"  of  the  procurement  research 
taxonomy  that  offer  the  greatest  opportunity 
for  cost  savings  and  Improvement.  An  analysis 
at  the  "issue"  level  of  the  procurement  re¬ 
search  taxonomy  could  identify  those  areas 
that  are  costly  to  implement,  difficult  to 
administer,  and  subject  to  frequent  delay,  as 
well  as  those  areas  that  offer  the  greatest 
benefits  to  the  DOD,  the  public,  and  industry. 
A  priority  system  of  procurement  research 
issues  would  identify  those  .areas  that  should 
receive  the  most  research  emphasis.  Limited 
resources  could  be  applied  to  "issues"  from 
the  top  down  so  that  the  most  important  areas 
are  researched  first. 

SUMMARY 

The  result  of  this  effort  is  only  the  first 
step  toward  defining  and  structuring  procure¬ 
ment  research.  It  will  be  up  to  the  procure¬ 
ment  community  as  to  whether  this  initial 
effort  is  accepted  and  used.  This  definition 
and  taxonomy  offer  a  basis  which  researchers 
can  use  to  more  closely  define  procurement 
research  and  its  relationship  to  the  procure¬ 
ment  process,  while  the  algorithm  provides 
the  researcher  or  procurement  manager  with  a 
logical  process  to  evaluate  the  research 
effort  as  to  its  applicability  to  procurement. 

Lastly,  observations  on  "What  Procurement 
Research  Should  Be..."  and  reconnendations 
for  further  research  were  offered  to  the  pro¬ 
curement  community  as  a  means  of  accelerating 
the  evolutionary  process  of  procurement  re¬ 
search. 

FOOTNOTES 

lA  sequential  decision  making  process  or 
model . 

zThese  categories  were  adopted  from  the 
Strayer-Lockwood  taxonomy.  See  reference  3 
and  14  for  further  discussion. 

3The  research  for  this  study  was  conducted 
prior  to  the  publication  of  the  Sixth  DOD 
Procurement  Research  Symposium. 
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The  application  of  risk  analysis  by  the  Direc¬ 
torate  of  Pricing  under  the  Deputy  for  Procure¬ 
ment  and  Manufacturing  in  the  Aeronautical 
Systems  Division  is  discussed  in  this  paper* 

A  method  of  analyzing  FPIF  contracts  is  pre¬ 
sented  along  with  guidelines  for  choosing  share 
ratios  and  ceilings.  Problems  and  pitfalls 
encountered  during  the  implementation  are  dis¬ 
cussed  with  several  suggestions  for  future 
modifications . 

*This  work  was  sponsored  by  the  Air  Force 
Systems  Command,  the  Air  Force  Office  of  Scien¬ 
tific  Research,  and  the  Business  Research  Man¬ 
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INTRODUCTION 


During  the  development  and  procurement  phases 
of  acquiring  a  major  defense  weapon  system, 
many  decisions  must  be  made  concerning  its  per¬ 
formance,  cost  and  scheduling.  An  important 
aspect  of  this  decision  making  process  is  the 
analysis  of  uncertainty  which  exists.  Common 
approaches  to  analyzing  uncertainty  have  in  the 
past  focused  on  the  decision  maker’s  intuition, 
on  sensitivity  analysis  and  on  risk  analysis. 

The  focus  of  this  paper  is  the  analysis  of  risk 
inherent  in  the  projection  of  cost  involved  in 
a  contract.  The  cost  of  a  specific  contract 
and  not  the  cost  of  the  entire  system  acquisi¬ 
tion  is  discussed  here. 

When  the  term  risk  analysis  is  used,  three  types 
of  risk  are  generally  implied.  These  types  are 
technical  risk,  cost  risk,  and  schedule  risk. 

In  the  study  of  a  large  weapon  system,  all  of 
these  risks  should  be  analyzed  to  determine 
which  alternatives  should  be  chosen  in  order  to 
maximize  the  probability  of  having  a  successful 
program.  When  a  specific  contract  for  a  program 
is  being  negotiated,  the  primary  variable  of 
interest  is  the  cost  risk.  Important  to  note, 
however,  is  that  the  cost  is  not  independent  of 
the  amount  of  technical  and  schedule  risk.  The 
technical  and  schedule  risks  are  important  fac¬ 
tors  in  the  estimation  of  the  cost  risk  and 
hence  the  cost  which  should  be  negotiated. 

A  price  analyst  in  the  Aeronautical  Systems 
Division  (ASD)  has  the  responsibility  of  deter¬ 
mining  and  negotiating  a  fair  and  reasonable 
cost  and  profit  for  a  contract.  Many  of  the 
cost  elements  may  be  estimated  with  some  degree 


of  certainty  and  will  be  referred  to  as  non- 
random.  Examples  of  non-random  cost  include 
negotiated  overhead  rates,  wage  rates  and  cer¬ 
tain  routine  labor  costs.  Other  cost  elements 
will  not  be  known  or  identifiable  with  certain¬ 
ty  and  will  be  referred  to  as  random.  The  ran¬ 
domness  in  the  cost  elements  may  be  caused  by 
some  factor  affecting  cost  or  may  be  totally 
unexplainable.  Causes  of  randomness  in  cost 
elements  include  design,  labor  and  material 
uncertainties  concerning  costs  and  amounts. 

The  price  analyst  must  still  estimate  the  cost 
and  negotiate  a  price  for  the  contract.  The 
purpose  of  this  paper  is  to  explain  how  risk 
analysis  is  being  used  to  reflect  the  extent 
to  which  randomness  affects  total  cost. 

Random  factors  affecting  cost  are  events  which 
the  contractor  cannot  control  and  which  are 
known  (or  suspected)  to  impact  one  or  more 
elements  of  a  cost  of  the  contract.  Risk 
analysis  is  a  procedure  for  analyzing  how  ran¬ 
domness  affects  the  total  cost.  An  analyst 
must  identify  the  random,  uncontrollable  fac¬ 
tors  and  assess  the  probability  of  different 
events  occurring.  Then  using  risk  analysis, 
the  distribution  of  the  total  cost  is  obtained. 
Results  of  a  risk  analysis  may  be  useful  to  a 
price  analyst  in  several  ways.  First,  it  will 
help  to  show  possible  actual  costs  which  might 
occur  and  the  probability  that  they  will  occur. 
Second,  it  may  help  in  determining  the  type  of 
contract  to  offer.  Third,  expected  cost  to 
the  government  can  be  determined.  And  fourth, 
actual  cost  can  be  bounded  or  given  a  range 
over  which  it  will  most  likely  occur. 

The  remainder  of  this  paper  will  discuss  an 
implementation  of  risk  analysis  by  the  Direc¬ 
torate  of  Pricing  under  the  Deputy  for  Pro¬ 
curement  and  Manufacturing  in  the  Aeronautical 
Systems  Division  at  Wright-Patterson  AFB. 

First,  the  objectives  of  the  risk  analysis 
study  will  be  given  with  a  brief  description 
of  the  methods  being  used  before  this  under¬ 
taking.  Second,  a  description  of  the  system 
implemented  will  be  given.  Finally,  some  of 
the  pitfalls  and  problems  encountered  will  be 
discussed  along  with  planned  modifications 
that  can  be  added  to  alleviate  these  problems. 

OBJECTIVE  FOR  RISK  ANALYSIS 

As  in  any  risk  analysis,  the  primary  goal  is  to 
determine  the  distribution  of  the  total  cost  of 
a  contract.  In  order  to  do  this  it  is  neces¬ 
sary  to  combine  information  from  numerous 
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sources  including  the  contractor's  proposal, 
the  PCO,  DCAA,  DCAS  and  (or)  APPRO.  The  risk 
analysis  discussed  here  was  developed  for  the 
price  analyst  whose  responsibility  it  is  to 
analyze  all  of  the  above  sources  of  informa¬ 
tion  and  to  negotiate  a  fair  and  reasonable 
price.  Before  the  negotiations  start,  an  ini¬ 
tial  position  and  objective  must  be  establish¬ 
ed  by  the  analyst  summarizing  all  of  the  cost 
subcomponents  and  the  total  price.  Histori¬ 
cally,  target  costs  were  established  by  sum¬ 
ming  the  most  likely  costs  for  each  of  the 
cost  subcomponents.  The  risk  analyses  per¬ 
formed  by  the  price  analyst  were  achieved  by 
summing  the  extreme  costs  for  each  of  the  cost 
subcomponent.  These  procedures  give  bad  esti¬ 
mates  of  both  the  most  likely  total  cost  (the 
cost  with  the  highest  probability  of  occurring) 
and  the  extreme  value  of  total  cost.  This  is 
because  the  sum  of  most  likely  values  of  sub¬ 
components  generally  is  not  the  most  likely 
value  of  the  total  cost  and  the  sum  of  extreme 
values  to  estimate  the  extreme  value  of  total 
cost  assumes  uncertainty  among  subcomponents 
is  perfectly  correlated. 

The  constraints  invoked  for  the  development  of 
the  risk  analysis  to  be  used  by  price  analysts 
were  as  given  below. 

1.  The  procedure  must  be  simple  and  easily 
performed . 

2.  The  information  needed  should  be  readily 
available  to  the  price  analyst. 

3.  The  output  should  be  easily  understood. 

4.  The  procedure  should  be  adaptable  to 
most  of  the  contracts  under  negotiation. 

5.  Computations  should  be  quick  and  easily 
made. 

In  order  to  make  the  procedure  adaptable  to 
the  contracts  under  negotiations  a  common  cost 
breakdown  was  needed.  It  was  found  that  the 
one  common  breakdown  was  that  given  by  Form 
DD633  to  provide  a  standard  format  which  con¬ 
tractors  submit  to  the  government  summarizing 
incurred  and  estimated  costs.  This  total  cost 
breakdown  was  into  the  categories  or  subcom¬ 
ponents  : 

a)  Material  (MAT) 

b)  Material  Overhead  (MATOH) 

c)  Interdivision  Transfer  (IT) 

d)  Direct  Engineering  Labor  (DEL) 

e)  Engineering  Overhead  (EOH) 

f)  Direct  Manufacturing  Labor  (DML) 

g)  Manufacturing  Overhead  (MOH) 

h)  Other  Costs  (OC) ,  and 

i)  General  and  Administrative  Expenses  (GAE) . 


In  evaluation  of  a  contract,  each  of  these  sub¬ 
components  are  usually  broken  down  further  and 
are  commonly  interrelated  as  shown  below,  where 
PI  through  P4  are  specific  percentage  figures, 
and  Rl  and  R2  are  specific  wage  rates.  The  * 
is  used  to  denote  multiplication. 

MAT  =*  Estimated  Material  Cost 

MATOH  *  PI*  MAT  +  Estimated  Independent 
Material  Overhead 

IT  -  Estimated  IT  Cost 

DEL  ■  (Estimated  Engineering  Hours)*  Rl 

EOH  =  P2*DEL  +  Estimated  Independent 
Engineering  Overhead 

DML  *  (Estimated  Manufacturing  Hours) *R2 

MOH  *  P3*DML  +  Estimated  Independent 
Manufacturing  Overhead 

OC  *  Estimated  Other  Cost 

SUBTOTAL  =  ST  *  MAT  +  MATOH  +  IT  +  DEL  +  EOH 
+  DML  +  MOH  +  OC 

GAE  =  P4*ST 

TOTAL  COST  *  TC  *  ST  +  GAE 

The  relationships  above  define  a  general  cost 
model  for  which  eight  estimates  are  needed  to 
determine  the  total  cost.  A  form  for  organi¬ 
zing  the  collection  of  data  required  for  the 
risk  analysis  is  given  in  Form  I.  The  esti¬ 
mates  requiring  minimum,  most  likely,  and  maxi¬ 
mum  values  must  be  supplied  by  the  analyst. 

For  each  of  the  cost  categories,  either  the 
cost  ($)  or  hours  must  be  estimated.  The  over¬ 
head  categories  are  divided  into  two  parts, 
the  independent  overhead  cost  and  the  overhead 
rate.  The  independent  overhead  cost  is  a  cost 
which  does  not  change  when  the  direct  cost 
changes.  Usually,  the  uncertainty  in  the  in¬ 
dependent  overhead  cost  is  due  to  future  busi¬ 
ness  conditions.  The  independent  overhead  cost 
is  commonly  allocated  to  the  direct  cost  and 
then  lumped  with  the  overhead  rate.  However, 
it  should  be  kept  separate  for  a  risk  analysis. 
The  overhead  rate  should  reflect  those  costs 
which  are  directly  proportional  to  the  direct 
cost.  It  is  assumed  here  that  this  rate  is 
known  with  certainty.  Using  the  Beta  distri¬ 
bution  implicitly  assumes  that  the  possible 
outcomes  can  be  bounded  in  some  finite  range. 

For  mature  systems,  this  is  not  an  unreasonable 
assumption. 

The  formula  for  the  mean  (expected)  and  vari¬ 
ance  are  theoretically  based  on  the  properties 
of  the  Beta  distribution  and  have  been  widely 
used  In  statistics,  risk  analysis  and  scheduling 
(PERT).  The  formulas  for  estimating  the  Mean 
and  Variance  for  a  Beta  distribution  are: 

Mean  -  L  +  4ML  4*  H 

,  6 
and 

Variance  ■ /H-LN 

(  6  ;  , 


98 


where  H  *  maximum,  L  3  minimum  and  ML  3  most 
likely.  These  calculations  are  actually  ap¬ 
proximations  and  are  very  good  if  the  distri¬ 
bution  is  not  too  severely  skewed. 


SUBCOMPONENTS 

MATERIAL 

MATERIAL  OVERHEAD 
RATE  FOR  MATERIAL 
INTERDIV  RNSERS 
DIRECT  ENGRG  LABOR 
WAGE  RATE 
NEGRG  OVERHEAD 
RATE  FOR  ENGRG 
DIRECT  MFG  LABOR 
WAGE  RATE 
MFG  OVERHEAD 
RATE  FOR  MFG 
THER  COSTS 
CSA  EXPENSES 

PERCENT  OF  SUBTOTAL 


FORM  I 

ESTIMATES  FOR  RISK  ANALYSIS 


ESTIMATES 


COST 

INDEPENDENT 

PI 


COST 

HOURS 


MOST 


MINIMUM  LIKELY  MAXIMUM 

-*-•  _ 

r 

The  risk  analysis  was  the  application  of  some 
well  known  statistical  analysis  to  the  cost 
model  described  above.  The  statistical  analy¬ 
sis  approach  is  useful  in  the  handling  of  con¬ 
tinuous  distribution  problems. 

The  statistical  concept  used  in  this  paper 
is  one  which  states  that  the  sum  of  Indepen¬ 
dent  random  variables  will  be  approximately 
normally  distributed  with  a  mean  equal  to  the 
sum  of  the  individual  means  and  variance  equal 
to  the  sum  of  the  individual  variances. 

If  we  rewrite  our  total  cost  model,  we  can  get 
it  into  the  form  needed.  That  is: 

TC  -  {(1+P1)*V1  +  V2  +  V4*(R1  +  Rl*P2)  +  V6* 
(R2  +  R2*P3)  +  V5  +  V3  +  V7  +  V8}*(l+P4) 

where  VI  =  Material  Cost 

V2  *  Independent  Material  Overhead 

V3  -  IT  Cost 

V4  »  Engineering  Hours 

V5  *  Independent  Engineering  Overhead 

V6  *  Manufacturing  Hours 

V7  =  Independent  Manufacturing  Overhead 

V8  *  Other  Costs 


Using  the  notation  E(V1)  to  represent  the  mean 
of  variable  VI  and  Var  (VI)  to  denote  the 
variance  of  VI,  the  mean  E(TC),  and  variance, 

Var  (TC),  of  the  total  cost  are: 

E(TC)  =  { (1+P1)*E(V1)  +  E(V2)  +  E(V4)*(R1+R1*P2) 
+  E (V6) * (R2+R2*P3)  +  E(V5)  +  E(V7)  + 
E(V8) }*(1+P4) 

Var  (TC)  =•  {(1+P1)2*V(V1)  +  Var  (V2)  +  Var(V4)* 
(R1+R1*P2) 2  +  Var  (V6)*(R2+R2*P3)2 
+  Var (V5)  +  Var(V3)  +  Var(V7)  +  Var 
(V8) }*(1+P4) 2 

The  distribution  of  the  total  cost  would  thus 
be  normal  with  mean,  E(TC)  and  variance,  Var 
(TC). 

The  normal  distribution  has  the  property  that 
probability  statements  can  be  made  using  only 
the  mean  and  variance.  For  instance,  there  is 
a  68%  chance  that  the  total  cost  will  be  be¬ 
tween  E(TC)  +  Var(TC)  and  a  95%  chance  that 
the  total  cost  will  be  between  E(TC)  + 

2  *  Var  (TC). 
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Once  the  mean  and  variance  of  the  total  cost 
have  been  determined,  the  distribution  of  the 
total  cost  is  completely  described  since  the 
total  cost  is  normally  distributed.  The 
functional  form  of  the  normal  distribution  is: 

f(c)  =  1  (c  -  U)2 

vHa  e  2cr 

Where  u  =  E(TC)  and  a2  =  Var(TC).  This  dis- 
tribution  is  then  the  completion  of  the  risk 
analysis . 

Let  us  now  investigate  the  effect  of  risk  on 
a  Fixed  Price  Incentive  Firm  (FPIF)  contract 
which  is  commonly  used  for  contracts  which 
have  some  risk  but  not  enough  to  resort  to  a 
cost  plus  type  contract.  In  particular,  the 
expected  profit  and  the  expected  cost  to  the 
government  will  be  expressed  in  terms  of  the 
following  variables: 

TC  «  Target  Cost 
TP  -  Target  Profit 
PC  38  Ceiling  Price 
a  *  Contractor's  share  of  underruns 
b  *  Contractor’s  share  of  overruns 
CC  *  Point  of  total  assumption  = 

PC  -  (TC  ±  TP)  +  TC 
1-b 

and 

f(c)  =  distribution  of  total  cost 

A  profit-cost  graph  for  a  Fixed  Price  Incen¬ 
tive  Firm  contract  is  presented  in  Figure  I. 
The  line  segments  to  the  left  of  TC,  between 
TC  and  CC  and  to  the  right  of  CC  are  given 
in  terms  of  the  cost  (C)  as 

LI  =*  TP  +  a(TC  -  C) 

L2  =  TP  -  b(C  -  TC)  and 
L3  “  PC  -  C 

Thus,  the  expected  profit  is  given  by: 

TC  CC 

E (Profit)  =  /  LI  f (c)dc  +  /  L2  f(c)  dc  + 

TC  TC 

TC+3a 

/  L3  f (c ) dc 
CC 

Of  course  if  CC  is  greater  than  TC  +  30,  then 
the  last  term  is  dropped. 

The  expected  cost  to  the  government  is  then 
E (TC)  +  E  (Profit). 


of  that  range. 


A  computer  program  was  provided  to  the  price 
analyst  for  the  calculations  necessary  for  com¬ 
paring  different  incentive  plans.  This  pro¬ 
gram  accepts  as  input  TC,  TP,  a,  a,  b,  and  PC, 
and  prints  the  expected  profit  and  cost  to  the 
government.  Also  the  high  and  low  profits  and 
cost  to  the  government  are  printed.  The  com¬ 
puter  program  use  Gaussion  Quadrature  as  a 
numerical  integration  technique  for  calcula¬ 
tion  of  the  expected  profit. 

If,  rather  than  evaluating  different  incentive 
plans,  one  wishes  to  use  the  risk  analysis  to 
establish  targets,  ceilings  and  share  ratios, 
Figure  II  demonstrates  how  this  might  be  done 
for  a  FPIF  type  contract.  The  two  points  TP 
(target  profit)  and  WP  (warrented  profit)  must 
be  supplied  by  the  price  analyst.  In  Figure  II, 
TC  is  the  target  cost  and  RAC  is  the  risk 
analysis  cost.  With  this  type  of  incentive 
contract  the  government  shares  in  overruns  and 
underruns  due  to  the  randomness  involved. 


FIGURE  II 


INCENTIVE  CONTRACT 


E (TC)  E(TC)  +  3  /Var  (TC) 


Notice  that  the  Normal  distribution  has  been 
truncated  at  plus  or  minus  3a  since  it  is  ex¬ 
tremely  unlikely  that  the  cost  will  be  outside 


TOO 


(See  next  page) 


OBSERVATIONS,  PITFALLS,  AND  FUTURE  MODIFICATIONS 


A  computer  program  is  currently  being  used  by 
price  analysts  for  performing  the  arithmetic 
needed.  The  output  of  this  program  is  illus¬ 
trated  in  Exhibit  I. 


In  the  implementation  of  the  risk  analysis  de¬ 
scribed  above,  the  author  found  several  prob¬ 
lems  which  were  difficult  to  overcome  and  may 
still  exist.  Each  of  these  problems  are  dis¬ 
cussed  below. 

1.  The  price  analyst  had  preconceived  defini¬ 
tions  of  terms  which  had  different  statis¬ 
tical  definitions.  For  instant,  expected 
cost  in  statistics  is  a  technical  term 
with  a  different  meaning  than  the  most 
likely  cost  or  mode.  The  definition  of 
these  terms  is  different  than  a  laymen's 
interpretation  and  therefore  created  a 
great  deal  of  confusion. 


» 


EXHIBIT  I 


ELEMENTS 

MINIMUM 

MOST 

LIKELY 

MAXIMUM 

MATERIAL 

COST 

8400.00 

9000.00 

12000.00 

MGT  OVERHEAD 

RATE  FOR  MATERIAL 

INDEPENDENT 

0.050 

0.00 

0.00 

0.00 

INTERDIV  TRSFRS 

COST 

1700.00 

1800.00 

2300.00 

DIRECT  ENGRG  LABOR 
WAGE  RATE 

HOURS 

11.500 

85.00 

95.00 

115.00 

ENGRG  OVERHEAD 

RATE  FOR  ENGRG 

INDEPENDENT 

0.700 

0.00 

0.00 

0.00 

DIRECT  MFG  LABOR 
WAGE  RATE 

HOURS 

11.000 

200.00 

230.00 

290.00 

MFG  OVERHEAD 

RATE  FOR  MFT 

INDEPENDENT 

1.700 

0.00 

0.00 

0.00 

OTHER  COST 

G&A  EXPENSE 

COST 

0.1000 

400.00 

450.00 

500.00 

SUMMARY,  CEILING/SHARE  COMPUTATION 


SUMMARY,  MINIMUM  COST 
SUMMARY,  MOST  LIKELY  COST 
SUMMARY,  MAXIMUM  COST 
EXPECTED  TOTAL  COST,  E(TC) 

RISK  ANALYSIS  COST,  RAC 

WARRANTED  PROFIT 
TARGET  PROFIT 
CEILING  PRICE 

PERCENT  DIFFERENCE  BETWEEN  RAC  AND  OBJECTIVE 


20373.93 

22427.07 

28887.37 

23161.60  EXCEEDED 
W/PROB  OF  50% 

25149.55  EXCEEDED 
W/PROB  OF  1%  OR  LESS 

1802.47 

2779.39 

27535.58 

11.77% 


SHARING  COMPUTATION: 


WGM  PROFIT  LESS  WARRANTED  PROFIT 
RISK  ANALYSIS  COST  LESS  OBJECTIVE  COST 
CONTRACTORS  SHARE 
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993.37 

2588.34 

38.36% 


2. 


Since  subjective  estimates  of  the  costs 
were  the  basis  of  the  risk  analysis,  biased 
positions  were  often  taken.  These  biases 
were  often  due  to  personal  feeling  towards 
what  would  happen  in  the  future. 

3.  There  tended  to  be  an  unconscious  attempt 
to  estimate  costs  which  could  be  negoti¬ 
ated  and  not  actual  cost.  This  is  very 
understandable  since  the  primary  job  of 
the  price  analyst  is  to  negotiate  the  con¬ 
tract. 

4.  Once  positions  and  objectives  had  been 
established  and  the  negotiations  began, 
adjustments  to  the  inputs  to  the  risk  anal¬ 
ysis  will  most  likely  not  be  made  because 
of  the  general  procedures  followed. 

5.  Target  profit  was  often  determined  using 
profit  weighted  guidelines  which  add  a 
percentage  factor  because  of  risk. 

6.  Future  changes  to  the  contract  and  other 
contractor  or  government  actions  tended  to 
make  it  difficult  to  define  exactly  what 
total  cost  was  being  estimated. 

7.  The  price  analyst  received  point  estimates 
of  most  of  the  cost  subcomponents.  The 
extent  of  the  randomness' was  subjectively 
evaluated.  Now  that  a  risk  analysis  is 
being  performed,  hopefully  more  than  point 
estimates  can  be  requested  and  obtained 
from  persons  furnishing  the  information  to 
the  price  analyst. 

Using  the  current  method  of  risk  analysis, 
three  major  assumptions  are  made  which  may 
cause  errors  in  the  prediction  intervals  if 
violated.  Each  of  these  are  discussed  below 
with  the  method  by  which  they  are  to  be  avoid¬ 
ed  in  the  future. 

1.  A  major  concern  in  this  analysis  is  an 

assumption  that  the  cost  subcomponents  are 
independent.  In  order  to  avoid  this  as¬ 
sumption  a  different  cost  breakdown  is 
needed  where  all  of  the  costs  are  indepen¬ 
dent  or  an  estimate  of  the  covariance  is 
needed.  The  later  is  not  possible  because 
of  the  bulk  of  information.  A  different 
cost  breakdown  might  be  possible  for  a 
specific  contract,  however,  this  would 
mean  that  each  contract  would  require  a 
different  cost  model  and  would  require  a 
great  deal  of  statistical  and  mathematical 
sophistication  on  the  part  of  the  price 
analyst.  The  analyst  has  supplied  upper 
limits  for  each  cost  subcomponents  and  the 
sum  of  the  maximums  is  printed  on  the  out¬ 
put  as  summary  maximum  cost.  This  would 
be  the  upper  limit  on  total  cost  if  there 
is  perfect  correlation  between  the  cost 
subcomponents.  At  the  other  extreme  of  no 
correlation  is  the  risk  analysis  cost  also 
printed  on  the  output.  In  order  to 


compensate  for  the  dependence  of  the  cost 
subcomponents,  these  two  extremes  of  per¬ 
fectly  correlated  deviations  and  indepen¬ 
dent  random  cost  subcomponents  can  be 
analyzed  and  adjustments  made. 

2.  In  order  to  have  a  normal  distribution  of 
the  total  cost,  the  cost  subcomponents  can 
not  be  dominated  by  a  single  cost  element. 

In  the  event  that  this  is  the  case  the  sum 
of  the  maximum  of  each  cost  subcomponent 

is  again  better  than  the  risk  analysis  cost. 

3.  The  current  system  assumes  that  overhead 
rates  and  wage  rates  are  fixed  constants. 

The  only  way  to  avoid  this  in  the  current 
system  is  to  run  the  risk  analysis  with 
different  values  and  compare  the  results. 

Currently,  plans  are  being  made  to  change  the 
procedure  being  used  to  one  which  will  assume 
that  the  total  cost  is  Beta  distributed  and 
allow  for  randomness  in  the  overhead  and  wage 
rates.  Note  that  this  will  eliminate  the  last 
two  assumptions  completely.  It  is  the  opinion 
of  the  author  that  neither  the  information 
needed  nor  the  sophistication  required  will  be 
available  to  the  price  analyst  for  handling 
the  interdependence  of  cost  subcomponents  in 
the  near  future.  These  dependencies  should  be 
recognized  by  the  price  analyst  and  adjusted 
for. 

SUMMARY 

The  application  of  risk  analysis  at  ASD  pricing 
has  been  very  successful.  The  author  feels 
that  an  indicator  of  this  success  is  the  fact 
that  the  price  analyst  is  now  starting  to  think 
in  a  manner  which  does  not  solely  focus  on 
point  estimates  but  allows  for  a  consideration 
of  the  risk  involved.  This  must  be  the  first 
step  and  then  more  sophisticated  procedures 
can  be  implemented.  With  this  interest  will 
come  a  request  for  information  in  a  different 
format  than  previously  requested  allowing  for 
more  accuracy  in  the  risk  analysis.  Also 
recommended  is  the  coordination  of  efforts  of 
ASD  pricing  with  those  who  are  considering  the 
risk  associated  with  other  elements  of  the 
program. 


* 


w 


* 


102 


BIBLIOGRAPHY 


1.  Black,  R.,  "Beta  Distribution  from  Moments," 

Grumman  Aerospace  Corporation,  PDM-OP-79-11 5,  June,  1979 

2.  Dienemann,  P.F.,  "Estimating  Cost  Uncertainty  Using  Monte 
Carlo  Techniques,"  RM-4854-PR,  The  Rand  Corporation.  January 

3.  Fisher,  G.H.,  Cost  Considerations  in  Systems  Analysis,  American 
Elsevier  Publishing  Company,  New  York 

4.  Klein,  Michael  R.,  Treating  Uncertainty  in  Cost  Analysis:  The  Beta 
Distribution,  U.S.  Navy  CNO  Resource  Analysis  Group 

5.  Murphy,  E.L.,  Jr.,  "Statistical  Methods  of  Measuring  the  Uncertainty 
of  Cost  Estimates,"  AD  718862,  February,  1970 

6.  Quade,  E.S.  and  Boucher,  W.I.,  System  Analysis  and  Policy  Planning-- 
Appl i cation  in  Defense,  American  Elsevier  Publishing  Company,  New  York 

7.  Schaefer,  D.  F. ,  Husic,  F.J.  and  Gutowski  M.F.,  "A  Monte  Carlo  Simulation 
Approach  to  Cost— Uncertainty  Analysis,"  AD  850054,  March,  1969 

8.  Wilder,  J.J.  An  Analytical  Method  of  Cost  Risk  Analysis,  Gruman  Aerospace 
Corporation,  PDR-0P-T77-12,  June,  1977 

9.  Worm,  George  H.,  "Application  of  Risk  Analysis  in  the  Acquisition  of  Major 
Weapon  Systems"  AF  Business  Research  Management  Center,  Wri ght-Patterson 
AFB,  Ohio,  1980 


THE  EVOLUTION  OF  RISK  AND  UNCERTAINTY 
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ABSTRACT 

Uncertainty  analysis  is  being  applied  to  an 
increasingly  larger  number  of  DOD  projects. 

The  starting  point  for  the  Army  Procurement 
Research  Office  (APRO)  has  been  to  develop  the 
concepts  and  factors  which  are  important  in 
its  uncertainty  analysis.  Present  work  is 
concerned  with  locating  appropriate  techniques 
and  automated  procedures  to  implement  some  of 
the  original  procurement  research.  The  pur¬ 
pose  of  this  article  is  to  present  some  of  the 
historical  work  leading  to  the  use  of  uncer¬ 
tainty  analysis  at  APRO  and  to  cite  specific 
references  which  will  be  used  on  future  appli¬ 
cations. 

A  STARTING  POINT:  APRO  MISSION 

The  mission  of  the  Army  Procurement  Research 
Office  (APRO)  is  to  conduct  procurement 
research  studies  leading  to  the  improvement  of 
Amy  procurement  management,  to  develop  new 
procurement  concepts  and  techniques,  and  to 
provide  consultation  services  on  procurement 
and  procurement  related  functions.  The  nature 
of  procurement  research  has  often  led  research¬ 
ers  to  use  qualitative  approaches  in  their 
studies.  The  application  of  rigorous  analyti¬ 
cal  tools  has  only  recently  been  used  by  the 
procurement  researchers. 

THE  TRANSITION:  THE  RECENT  STORY 

Much  of  the  past  work  of  the  APRO  has  laid  a 
conceptual  groundwork  for  future  uncertainty 
analysis.  This  work  has  been  concerned  with 
establishing  relevant  criteria  and  the  analysis 
of  individual  programs  (Should  Cost  Analysis, 
evaluation  of  competitive  strategies,  to  name 
a  few).  This  has  led  to  a  greater  understand¬ 
ing  of  the  important  factors  which  will  be 
included  in  future  uncertainty  analyses.  By 
understanding  the  Interrelationships  amongst 
various  factors,  the  procurement  community  has 
begun  to  more  effectively  articulate  the  pro¬ 
blems  and  desires  of  uncertainty  analysis. 
However,  past  efforts  have  centered  more  on 
establishing  a  cause/effect  relationship  with 
the  goal  of  finding  relationships  and  methods 
which  reduce  or  eliminate  the  uncertainty. 
Whereas  these  studies  have  recognized  that 
complex  trade-offs  had  to  be  made,  little 
effort  was  expended  In  developing  methodologies 
for  this  trade-off  analysis. 


^Numbers  In  parentheses  denote  the  number 
referenced  in  the  list  of  references. 


More  recently,  the  procurement  community  is 
becoming  more  receptive  to  the  concept  of  un¬ 
certainty  analysis.  For  example,  the  APRO 
Study  806-1,  Relating  Acquisition  and  Contract 
Planning,  completed  in  June,  1979  (10)*,  illus¬ 
trates  a  conceptual  framework  which  would  be 
amenable  to  uncertainty  analysis  using  Monte 
Carlo  simulation  to  determine  uncertainty  in 
various  procurement  alternatives.  In  November 
1979,  A  Summary  and  Analysis  of  Army  Project 
Manager  Responses  to  a  Questionnaire  for  the 

Army  Acquisition  Task  Force  (ASA-RDA) ,  (3)  was 
written.  This  publication  contains  a  decision- 
tree  model  which  could  be  automated  and  supple¬ 
mented  with  utility  assessment  capability,  re¬ 
sulting  in  a  model  for  evaluating  uncertainty 
in  acquisition  strategy  planning.  This  publi¬ 
cation  indicated  that  the  model  contained  there¬ 
in  would  be  utilized  for  a  current  APRO  study 
project  904,  Acquisition  Strategy  Development. 

The  above  information  is  in  no  way  exhaustive 
of  all  the  APRO  literature  involved  in  the 
transition  towards  the  use  of  uncertainty 
analysis.  However,  it  does  show  the  evolution 
of  the  interest  of  the  procurement  research 
community  from  the  historical  quantitative 
study  to  the  current  conditional  studies,  and 
eventually  to  the  uncertainty  analysis  itself. 
However,  research  on  the  analytical  tools  is 
one  current  concern. 

The  research  at  APRO  is  now  concerned  with 
finding  contemporary  examples  of  uncertainty 
analysis  which  would  help  in  the  theoretical 
validation  of  any  model  which  APRO  may  develop. 
A  March  1976  article  by  the  Aerospace  Industries 
Association  (AIA) ,  titled  Risk  Elements  in 
Government  Contracting,  (13)  provides  a  good 
discussion  on  risk  elements  in  procurement 
alternatives.  Additionally,  Decisions  and 
Designs,  Inc.  has  developed  an  extensive  capa¬ 
bility  in  Decision  Analysis  and  Utility  Theory 
as  evidenced  by  studies  of  the  Multi-Attribute 
Analysis  of  Alternative  Naval  Aviation  Plans 
(7),  An  Analysis  of  Alternative  Mideastern  Oil 
Agreements,  (1)  and  many  studies  using  Multi- 
Attribute  Utility  Theory,  (2,  4,  15)  to  name 
only  a  very  few.  Through  similar  studies,  the 
use  of  the  Mult I- At tribute  Utility  Model  (MAUM), 
trade-off  analysis,  and  computer  aided  analysis 
has  become  more  acceptable  by  reducing  the 
manual  manipulation  and  technical  expertise 
needed  to  apply  the  models  (14).  This  Is  in 
sharp  contrast  to  former  studies  which  needed 
simple  relationships  which  all  could  understand 
before  it  was  considered  acceptable. 
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the  acceptability  and  use  of  uncertainty  analy¬ 
sis,  consideration  should  be  given  to  the  work 
done  by  the  Social  Science  Research  Institute, 
University  of  Southern  California  (5,  9,  12). 
This  work  has  helped  APRO  to  gain  a  better 
understanding  on  topics  such  as  group  utility 
assessing  and  evaluation  methodology. 

It  also  appears  that  the  past  effort  was  ham¬ 
pered  by  a  reluctance  to  accept  uncertainty 
analysis  coupled  with  the  complexity  and 
manual  manipulation  of  data  requirements.  Much 
of  the  recent  work  has  made  significant  pro¬ 
gress  in  making  the  products  more  user  oriented 
through  computer  assisted  models.  Perhaps  by 
this  very  act  of  placing  the  manager  in  a  role 
of  model  developer  while  also  taking  away  the 
technical  requirements  of  evaluation,  we  have 
gained  more  acceptability.  It  is  particularly 
interesting  to  note  that  many  of  the  models 
developed  by  DDI  make  extensive  use  of  computer 
aids  (see  reference  4  for  illustrations  of 
computer  aids). 

It  is  anticipated  that  in  early  1981,  APRO  will 
begin  a  study  called  "Uncertainty  Analysis  of 
Acquisition  Alternatives."  The  study  most 
likely  will  incorporate  subjective  and  object¬ 
ive  data  and  will  utilize  a  Multi-attribute 
decision  model.  It  is  expected  that  this  model 
will  be  automated  to  provide  the  user  a  com¬ 
puter  assisted  decision  aiding  model.  APRO 
80-10,  Contractor  Furnished  Equipment  v.  Govern¬ 
ment  Furnished  Equipment  -  An  Alternative 
Analysis ,  will  address  some  of  the  subjective 
factors  in  the  CFE  vs.  GFE  analysis  process. 

A  decision  aiding  model  will  be  developed  to 
assist  in  the  decision  process,  of  choosing  the 
appropriate  acquisition  approach.  Also,  APRO 
is  serving,  on  a  consulting  basis,  on  the  Multi¬ 
ple  Launch  Rocket  System  (MLRS)  Study.  APRO 
was  asked  by  the  study  team  to  recommend  a  pro¬ 
bability  distribution  for  recurring  cost  sav¬ 
ings  under  competition.  APRO  analyzed  the  data 
assembled  and  reported  by  the  Analytic  Sciences 
Corporation  in  their  report  titled,  "Predicting 
the  Costs  and  Benefits  of  Competitive  Produc¬ 
tion  Sources"  (31  Dec  79) . 


APRO  806-1  (10)  provides  a  starting  point  for 
future  APRO  studies  using  uncertainty  analysis. 
Figure  1  below  illustrates  the  concept  of  de¬ 
composition  which  Rex  Brown  discusses  in 
Research  and  the  Credibility  of  Estimates  (11) . 
Whereas  the  individual  may  not  be  able  to  give 
an  assessment  of  the  uncertainty  of  an  alter¬ 
native  course  of  action,  the  decomposition 
method  provides  the  analyst  with  a  systematic 
method  of  partitioning  the  problem  into  succes¬ 
sively  more  specific  sub-elements  until  a  highly 
specific  level  of  problem  definition  is  reached. 
These  sub-elements  are  then  combined  into 
increasingly  aggregated  measures  based  upon 
predicted  system  characteristics  and  finally 
aggregated  to  the  estimate  of  an  alternative’s 
utility.  The  approach  which  has  been  discussed 
above  is  commonly  called  the  multi-attribute 
utility  approach  (2) . 

The  structure  of  the  model  makes  clear  all  of 
the  elements  and  their  relationships  with  each 
other.  If  any  disagreement  on  the  structure  is 
found,  it  is  fairly  easy  to  enter  the  considera¬ 
tion  into  the  model,  change  values,  or  remove 
elements  as  required.  Thus,  the  model  is  not 
only  the  analytical  structure,  it  also  serves 
the  important  role  of  facilitating  communica¬ 
tion  amongst  those  involved  in  the  decision 
making  process. 

The  specific  structural  principle  of  this  model 
is  the  hierarchical  decomposition.  Each  ele¬ 
ment  of  an  alternative  is  partitioned  into 
its  sub-elements,  each  of  which  are  partitioned 
into  its  sub-elements,  etc.  This  decomposition 
continues  until  the  final  parameters  affecting 
the  alternative  are  reached.  When  thus  iso¬ 
lated,  uncertainties  of  these  parameters  can  be 
assessed  by  groups  of  experts  and  recomposition 
(aggregation)  takes  place  in  the  inverse  order 
(6). 
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The  first  level  of  the  hierarchical  structure 
of  figure  1  corresponds  to  the  single  value 
representing  the  utility  of  each  of  the  alter¬ 
natives  to  be  analyzed.  The  second  level  re¬ 
presents  the  first  of  the  major  elements  of 
concern:  Domestic  or  Foreign.  Under  domestic, 
the  area  of  concern  is  performance  by  the  Army 
or  other  service.  Successive  levels  involve 
cost,  time,  performance,  and  other  considera¬ 
tions  (political,  social,  etc.).  These  consid¬ 
erations  are  decomposed  into  layers  considering 
marketplace,  program  resources,  etc.  When  the 
final  level  of  decomposition  has  been  achieved, 
assessments  of  the  outcomes  of  the  final  layers 
can  be  made.  This  decomposition  should  con¬ 
tinue  until,  when  an  individual  is  asked  to 
estimate  the  uncertainty,  a  reply  such  as  "it 
depends"  cannot  be  given.  The  "it  depends" 
answer  really  signifies  the  need  to  further 
decompose  the  problem  into  still  smaller  sub¬ 
elements.  These  types  of  questions  require  the 
use  of  conditioning  variables.  For  example, 
an  alternative  course  of  action  frequently 
depends  upon  the  dollar  amount,  a  conditional 
variable . 


However,  as  DDI  report  76-10  (14)  relates, 
numerous  considerations  will  impact  upon  the 
form  which  the  model  takes.  This  report  gives 
the  decision  analysis  users  a  way  to  identify 
appropriate  analytic  approaches  for  any  given 
decision  situation.  It  addresses  the  decision 
situation,  the  analytic  options  available,  and 
various  performance  measures.  It  would  be 
beneficial  to  use  existing  computer  programs 
and  methodology. 

In  addition  to  the  programs  addressed  therein, 
consideration  should  also  be  given  to  the  use 
of  RISCA  and  VERT,  available  through  the  Sys¬ 
tems  and  Cost  Analysis  Department  (SCAD),  Army 
Logistics  Management  Center  (ALMC) ,  Fort  Lee, 
Virginia. 

Another  possible  decision  model  for  APRO  studies 
would  involve  the  use  of  a  decision  diagram. 

The  Acquisition  Strategies  report  (3)  suggested 
a  model  which  would  form  the  basis  of  a  decision 
diagram  for  acquisition  strategy  planning.  The 
model  in  Figure  2  below  is  an  altered  form  of 
the  model  presented  in  the  APRO  report. 


The  Decision  Analysis  Handbook  (6)  illustrates 
the  decomposition  process.  It  discusses  rules 
of  recombination  (multiplicative  and  additive) 
and  the  steps  in  developing,  analyzing,  and 
validating  the  model.  When  the  APRO  uncer¬ 
tainty  model  is  fully  developed,  it  will  allow 
users  to  interact  with  a  computer  assisted 
decision  aiding  tool.  However,  to  implement 
this  tool,  certain  steps  remain  to  be  taken: 

1.  The  sub-elements  of  each  alternative 
have  to  be  identified. 

2.  Means  to  gather  subjective  and  objec¬ 
tive  data  must  be  provided. 

3.  The  computer  model  has  to  be  developed. 

4.  Implementation  and  use  of  the  model 
requires  briefings  and  training. 


This  type  of  model  distinguishes  actions  from 
events.  Following  a  specific  action,  one  or 
several  events  may  occur.  When  the  system  is 
developed,  the  final  objective  is  to  assess 
the  likelihoods  of  the  occurrences  of  each 
event.  Thus,  this  model  requires  a  considerable 
amount  of  judgmental  assessments.  Many  computer 
aided  analysis  programs  are  available  to  help 
with  the  final  evaluation  of  the  alternatives 
(8).  As  in  the  hierarchical  decomposition 
process,  this  method  attempts  to  decompose  the 
problem  into  smaller  sub-elements.  However, 
this  process  also  requires  assessment  of  like¬ 
lihoods  of  outcomes  (events) .  The  hierarchical 
decomposition  model,  on  the  other  hand,  requires 
the  assessments  of  weights  or  measurements  of 
importance  for  the  various  sub-elements.  Both 
methods  require  subjective  assessments  (pro¬ 
babilities  of  outcomes  or  events  or  assessment 
of  individual  parameters).  This  type  of  model 
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was  used  in  the  Design-to-Cost  evaluation  for 
the  Navy's  Electronic  Warfare  System  (2).  The 
Decision  Analysis  Handbook  (6)  provides  an 
excellent  discussion  on  this  decomposition  pro¬ 
cess  and  the  methodology  for  recomposition  (the 
average  out  and  fold  back  methodology). 

The  automation  of  the  process  facilitates  the 
testing  of  certain  "what  if”  questions  or  un¬ 
certainty  analysis.  For  example,  one  may 
desire  to  see  what  events  in  the  Acquisition 
Strategy  Planning  Process  are  more  important 
than  others.  The  analysis  may  indicate  that 
under  most  reasonable  circumstances,  certain 
events  will  have  little  to  no  effect  in  the 
planning.  This  will  help  management  to  put 
those  areas  of  lesser  interest  into  proper 
perspective,  leaving  more  time  for  evaluating 
those  aspects  of  most  importance.  This  model 
development  will  require  input  of  opinions 
from  many  individuals.  This  very  aspect 
appears  to  some  to  mean  that  the  model  is  not 
to  be  used  since  it  is  just  a  matter  of  opinion. 
The  virtue  of  the  model  is  that  it  is  system¬ 
atically  derived  and  logically  constructed, 
making  use  of  as  much  information  as  is  avail¬ 
able.  When  new  information  is  available,  it 
should  be  incorporated  where  possible  or  used 
to  judge  the  analytic  ability  of  the  model. 

To  be  successful,  the  application  of  decision 
analysis  to  major  DOD  projects  will  require 
several  elements  to  be  present: 


esoteric  analytical  methods  are  being  employed. 
Rather,  the  circumstances  of  the  problem  defini¬ 
tion/analysis  are  being  controlled.  This  could 
have  significant  effect  of  in-house  decision 
analyses:  success  or  failure  of  the  project  is 

dependent  upon  the  mechanics  of  application 
(team  concept,  elements  present,  etc.).  This 
is  not  to  say  that  sound,  rigorous  logic  and 
analysis  is  not  important.  Rather,  the  impli¬ 
cation  is  that  these  other  factors  are  as 
important  as  the  analytical  method. 

SUMMARY 

It  is  important,  in  summary,  to  reiterate  that 
APRO  studies  have  not  completely  addressed  all 
of  the  aspects  of  uncertainty  analysis.  Past 
efforts  have  laid  the  conceptual  ground  work 
which  future  efforts  will  use.  This  article 
has  presented  several  approaches  to  uncertainty 
analysis:  heirarchical  decomposition  and 
decision  diagrams  (also  called  trees)  .  The 
selection  of  the  technique  depends  upon  the 
specific  circumstance  being  analyzed.  Many 
examples  of  analysis  are  available.  This  arti¬ 
cle  has  attempted  to  list  several  which  will 
be  useful  to  future  APRO  studies.  Finally,  it 
appears  certain  that  uncertainty  analysis  on 
other  than  simple  problems  requires  the  use  of 
automated  analysis  procedures.  Many  of  the 
references  selected  for  inclusion  in  this  arti¬ 
cle  incorporate  or  address  computer  aided 
analysis . 


1,  A  problem  to  be  solved. 
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defined  roles  in  the  process.  Research  into 
the  applications  does  not  indicate  that 


2 .  An  Application  of  Multi-Attribute  Utility 

Theory:  Design-To-Cost  Evaluation  of  the  US 

Nsvy's  Electronic  Warfare  System,  Decisions  and 
Designs,  Inc.,  Michael  L.  Hays,  Michael  F. 
O'Connor,  Cameron  R.  Peterson,  October  1975. 

3.  "Acquisition  Strategies  in  Army  Programs", 

A  Summary  and  Analysis  of  Army  Project  Manager 
Responses  to  a  Questionnaire  for  the  Army  Acqui¬ 
sition  Task  Force  (ASA-RDA) ,  an  APRO  report 
dated  21  November  1979. 

4.  Decision  Analysis  as  an  Element  in  an 
Operational  Decision  Aiding  System,  Phase  II, 

Technical  Report  75-13,  prepared  for  Operation 
Decision  Aids  Project  (Code  455)  by  Decisions 
and  Designs,  Inc.,  R.V.  Brown,  C.R.  Peterson, 
W.H.  Shawcross,  and  J.W.  Ulvila,  authors.  Report 
prepared  in  November  1975  (DLSIE  LD  35297A)*. 


^Defense  Logistics  Studies  Information  Exchange 
(DLSIE),  US  Army  Logistics  Management  Center, 
Fort  Lee,  VA  23801 


107 


ft 


5 .  Experimental  Tests  and  Independence  Assump¬ 

tions  for  Risky  Multi-Attribute  References, 
Detlof  Von  Winterfeldt,  Scocial  Science  Re¬ 
search  Institute,  University  of  Southern  Cali¬ 
fornia,  October  1976,  (SSRI  Research  Report 
76-8). 

6.  Handbook  for  Decision  Analysis,  Decisions 
and  Designs,  Inc.,  Scott  Barclay,  Rex  Brown, 
Clinton  W.  Kelly,  Cameron  R.  Peterson,  Lawrence 
D.  Phillips,  Judith  Selvidge,  September,  1977. 
(Defense  Advanced  Research  Projects  Agency, 

ARPA  Order  3052) . 

7 .  A  Multi-Attribute  Utility  Approach  for 
Evaluating  Alternative  Naval  Aviation  Plans, 

Decisions  and  Designs,  Inc.,  Michael  F. 
O’Connor,  Thomas  R.  Rhees,  James  J.  Allen, 
September,  1976.  (ARPA  Order  No.  3052). 

8 .  A  Pilot  Survey  of  Computer  Programs  for 

Decision  Analysis,  Decisions  and  Designs, 

Inc.,  J.W.  Ulvila ,  January,  1975  (DLSIE 
LD  No.  34581A) . 

9 .  Rank  Weighting  in  Multi-Attribute  Utility 

Decision  Making;  Avoiding  the  Pitfalls  of 

Equal  Weights,  William  G.  Stillwell  and 
Ward  Edwards,  Social  Science  Research  Insti¬ 
tute,  University  of  Southern  California, 
September,  1979,  (SSRI  Research  Report  79-2). 


10.  Relating  Acquisition  and  Contract  Planning, 
Army  Procurement  Research  Office,  Robert  F. 
Williams  and  Wayne  V.  Zabel,  June  1979  (APRO 
806-1). 

11 .  Research  and  the  Credibility  of  Estimates, 
Rex  V.  Brown,  Division  of  Research,  Graduate 
School  of  Business  Administration,  Harvard 
University,  1969  (SBN  87584-079-5). 

12 .  Research  on  the  Technology  of  Inference 
and  Decision,  Ward  Edwards,  Richard  S.  John, 
and  William  Stillwell,  Social  Science  Research 
Institute,  University  of  Southern  California 
(SSRI  Research  Report  79-1),  January,  1979. 

13 .  Risk  Elements  in  Government  Contracting, 
Aerospace  Industries  Association,  March,  1976. 

14 .  Selecting  Analytic  Approaches  for  Decision 
Situations.  A  Matching  of  Taxonomities,  Rex  V. 
Brown  and  Jacob  W.  Ulvila,  Decisions  and 
Designs,  Inc.,  October,  1976  (DLSIE  LD  No. 
38616A) . 

15 .  On  Using  Scenarios  in  the  Evaluation  of 
Complex  Alternatives,  Michael  F.  O’Connor 

and  Ward  Edwards,  Decisions  and  Designs,  Inc., 
December,  1976. 


108 


DECISION  MAKING  IN  CONSTRUCTION  OPERATIONS 


Dr.  Richard  E.  Larew 
Construction  Engineering  and  Management 
Department  of  Civil  Engineering 
The  Ohio  State  University 
Columbus,  Ohio 


INTRODUCTION 

This  paper  is  based  upon  studies  of  construction 
operations  in  the  1970's,  as  well  as  studies 
currently  under  way  in  the  1980's.  Most  of 
these  studies  have  concerned  firms  with  annual 
sales  that  range  between  one  and  twenty  million 
dollars  per  year.  The  problems,  to  be  discussed 
in  detail,  will  concern  the  setting  of  a  price 
when  submitting  competitive  bids  for  the  perfor¬ 
mance  of  construction  work. 

The  author  has  not  made  extensive  studies  of 
specific  DOD  problems;  however,  based  on  brief¬ 
ings  from  government  agencies  and  construction 
firms,  as  well  as  studies  of  large-scale  prob¬ 
lems  on  both  buyer  and  seller  sides,  the  author 
has  concluded  that  lessons  learned  from  studies 
of  construction  operations  to  all  segments  of 
the  construction  industry,  to  buyers  as  well  as 
sellers,  and  of  course,  to  the  Department  of  De¬ 
fense's  problem  of  improving  decision  maxing  in 
major  acquisition  activities. 

Taking  these  conclusions  even  further,  it  is 
the  author's  belief  that  the  Department  of  De¬ 
fense  has  no  unique  problems  with  respect  to 
risk  and  uncertainty.  The  problems  encountered 
by  DOD  are  similar  to  those  encountered  by  other 
federal  agencies,  state  governments,  foreign 
governments,  large  buyers  of  construction  ser¬ 
vices,  and  large  contractors.  The  only  essen¬ 
tial  differences  are  the  specifics  of  the  ap¬ 
plication  and  the  values  of  the  parameters,  as 
we  transfer  technologies  that  have  been  useful 
in  other  areas  to  the  problems  of  DOD. 

The  three  major  objectives  of  this  paper  are: 

1.  To  report  some  methods  that  have  been  ap¬ 
plied  successfully  in  case  studies  of  con¬ 
struction  operations; 

2.  To  report  the  results  of  these  studies  and 
comment  upon  causes  of  major  problems  that 
have  been  encountered;  and, 

3.  To  offer  some  recommendations  for  improving 
decision  making  on  large  future  DOD  pro¬ 
jects. 

The  body  of  this  paper  is  divided  into  five  ma¬ 
jor  steps  for  improviding  decision  making  in 
construction:  (1)  Starting;  (2)  Baselining; 

(3)  Modeling;  (4)  Applying;  and  (5)  Evaluat¬ 
ing. 

Within  each  of  these  five  divisions,  the  author 
discusses  some  of  the  methods  used,  results 


obtained,  and  major  problems  encountered.  Fol¬ 
lowing  these  discussions,  the  author  offers 
some  final  recommendations  relative  to  the  im¬ 
provement  of  decision  making  on  large  DOD  pro¬ 
jects  . 

Finally,  the  author  has  included  a  number  of 
appendices  to  facilitate  the  transfer  of  our 
approach  at  Ohio  State  to  other  participants  in 
this  workshop.  Appendix  A  outlines  the  RS  dis¬ 
tribution  which  we  use  to  approximate  unimodal 
random  variables.  Appendix  B  describes  what  we 
call  the  M-Star  Methodology  for  finding  an  op¬ 
timum  bid  in  a  construction  market.  The  metho¬ 
dology  described  in  this  appendix  is  the  funda¬ 
mental  beginning  point  for  modeling  markets 
that  behave  in  a  simplistic  and  well-behaved 
manner.  The  final  appendix,  Appendix  C,  lists 
lambda  parameters  that  are  needed  for  the  im¬ 
plementation  and  use  of  the  RS  distribution. 


STEP  1:  STARTING 


When  beginning  a  study  of  competitive  bidding 
for  any  construction  program,  we  must  be  cer¬ 
tain  that  we  are  laying  a  solid  foundation  for 
acceptance  and  application  of  the  models  de¬ 
veloped.  The  four  most  critical  elements  at 
this  stage  are:  the  identification  of  the 
firm's  real  objectives;  the  securing  of  approv¬ 
al  and  support  for  the  study  from  the  chief 
executive  officer:  the  establishment  of  an 
agreed-upon  method  for  evaluating  results;  and 
the  identification  of  all  significant  con¬ 
straints. 

The  author  has  found  that  many  companies  claim 
to  have  one  objective  while,  in  fact,  the  real 
objective  goes  unment ioned .  In  the  general 
case  of  competitive  bidding,  we  suggest  that 
the  primary  objective  of  a  construction  com¬ 
pany  should  be  to  maximize  net  profits  in  dol¬ 
lars.  Yet  we  find  in  many  companies  that  this 
is  not  the  true  objective;  for  example,  a  com¬ 
pany  might  wish  to  maintain  a  maximum  level  of 
employment  for  nephews,  sons,  cousins,  and 
other  members  of  the  owner's  family.  The  true 
objective  of  the  company  must  be  determined  be¬ 
fore  one  proceeds  with  the  studies,  or  else 
the  policies  developed  will  not  achieve  the 
real  objectives  of  the  firm. 

In  working  with  some  government  agencies,  the 
author  has  found  that  the  true  objective  of  re¬ 
sponsible  persons  may  not  be  that  of  maximizing 
the  use  of  taxpayer  dollars.  For  example,  the 
true  objective  of  one  government  agency  with 
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which  the  author  has  worked  is  to  avoid  embar¬ 
rassment.  The  most  serious  embarrassment,  from 
this  agency’s  point  of  view,  is  having  to  go 
back  to  Congress  and  ask  for  more  money.  Its 
second  most  serious  embarrassment  is  being  guil¬ 
ty  of  not  spending  all  the  funds  that  were  al¬ 
located  for  a  given  project. 

The  important  point  to  remember  is  that  we  will 
develop  the  wrong  solution  if  we  fail  to  recog¬ 
nize  the  true  objective  of  the  agency  we  are 
trying  to  serve.  True  objectives  must  be  ham¬ 
mered  out  at  the  onset.  This  is  hard  work,  and 
can  take  several  weeks  to  accomplish. 

The  need  to  secure  the  approval  and  full  sup¬ 
port  of  the  chief  executive  offier  of  the  com¬ 
pany  prior  to  beginning  any  study  cannot  be 
overemphasized.  We  have  learned  this  the  hard 
way.  As  one  example,  in  a  study  that  was  fully 
approved  by  the  president  of  a  company,  a  one- 
year  investigation  resulting  in  recommendations 
for  considerable  capital  investments  was  scut¬ 
tled  at  9:00  am  on  the  day  on  which  it  was  sche¬ 
duled  to  be  presented  to  the  Board  of  Directors. 
The  cost  engineer  preparing  the  study  was.  In 
fact,  fired  at  that  time,  thus  preventing  the 
presentation  of  recommendations  to  the  Board 
later  on  that  day.  The  individual  in  question 
was  fired  by  the  Chairman  of  the  Board,  who  had 
never  been  included  in  the  problem  being  solved 
since  he  was  approaching  retirement.  The  Chair¬ 
man  of  the  Board  had  a  buy-sell  agreement  and, 
as  it  turned  out,  the  amount  of  money  he  would 
take  with  him  when  he  left  the  company  would  be 
severely  reduced  if  long-term  Investments  were 
entered  Into  prior  to  his  retirement.  Had  the 
Chairman  of  the  Board  been  brought  in  earlier 
and  had  we  insisted  upon  his  support  of  our 
studies  relative  to  long-term  capital  invest¬ 
ments,  it  would  have  been  apparent  that  he  was 
not  supportive  and  the  study  would  not  and 
should  not  have  proceeded. 

The  third  major  element  in  this  stage  is  to 
have  all  parties  agree  upon  a  method  of  eval¬ 
uating  the  results  of  any  study.  Our  experience 
has  been  that  contractors  will  buy  any  model  we 
give  them  if  they  win  the  first  two  or  three 
jobs  after  we  give  them  a  mathematical  model, 
and  they  will  reject  any  model  if  they  do  not 
win  the  first  two  or  three  jobs  after  we  give 
them  a  new  pricing  policy  —  that  is,  if  we 
have  not  previously  agreed  that  a  sample  size 
of  twenty  or  thirty  projects,  or  perhaps  more, 
would  be  collected  before  final  judgment  is 
pas  sed . 

The  last  major  concern  in  starting  any  study  is 
the  identification  of  all  major  constraints. 

This  may  sometimes  become  a  sensitive  matter  be¬ 
cause  contractors  do  not  want  to  disclose  the 
fact  that  they  really  don't  have  all  the  money 
in  the  world,  that  they  are  approaching  bank¬ 
ruptcy,  that  their  lines  of  credit  are  not  what 
they  are  claimed  to  be.  or  other  factors  that 
would  seriously  impact  decision  making.  It  is 
necessary  to  explore  sensitive  areas  before 


proceeding.  The  usual  constraints  in  competi¬ 
tive  bidding  include  such  factors  as  the  con¬ 
tractor’s  bonding  limit,  a  limit  on  his  abili¬ 
ty  to  prepare  and  submit  more  than  a  given  num¬ 
ber  of  bids  in  a  given  time  period,  and  the 
firm's  inability  to  execute  or  perform  more 
than  a  given  volume  of  construction  work  in  a 
given  time  period. 

There  are  many  other  factors  that  should  be 
and  are  considered  before  beginning  a  study. 

We  never  proceed  with  any  study  involving  risk 
or  uncertainty  in  a  construction  operation 
without  attending  to  the  four  major  items: 
identification  of  real  objectives,  securing 
the  support  of  the  chief  executive  officer, 
establishment  of  an  agreed-upon  method  for 
evaluating  results,  and  the  identification  of 
major  constraints. 

STEP  2:  BASELINING 


The  second  step  in  our  studies  is  to  establish 
a  baseline  against  which  we  can  measure  future 
results.  In  the  process  of  establishing  this 
baseline,  we  become  familiar  with  the  major 
variables  that  are  impacting  the  company  and 
the  decisions  that  must  be  made.  In  the  case 
of  competitive  bidding,  we  make  a  retrospective 
study,  often  going  back  two  or  three  years,  to 
determine  how  well  the  company  might  have  done 
in  its  market  had  it  had  perfect  knowledge  of 
that  market.  By  this  we  mean,  how  many  dol¬ 
lars  could  the  company  have  made  if  it  had 
known  exactly  the  bid  of  each  competitor  prior 
to  bidding.  The  major  problem  in  establishing 
this  baseline  is  that  many  companies,  although 
they've  been  in  business  for  many  years,  have 
no  records  at  all  of  previous  cost  estimates, 
bids  submitted  by  competitors,  or  other  facts 
of  importance.  It  is  sometimes  necessary  to 
halt  the  study  at  this  time  and  wait  for  six 
months  or  a  year  while  the  required  data  is 
collected . 

Based  upon  studies  of  a  large  number  of  opera¬ 
tions,  we  have  acquired  some  idea  of  the  level 
of  achievement  of  construction  firms  using 
their  existing  methods.  As  a  general  rule, 
the  typical  firm  will  yield  no  more  than  12 
to  25  percent  of  the  so-called  "dollars  in 
the  market."  That  is,  the  typical  firm  is 
able  to  earn  from  one-eighth  to  one-fourth 
the  the  dollars  the  firm  could  earn  if  it 
knew  the  exact  price  of  every  competitor  prior 
to  submitting  a  bid. 

When  firms  claim  higher  success  rates,  we  con¬ 
sistently  find  that  they  are  either  omitting 
those  embarrassing  cases  in  which  they  were 
so  far  out  of  line,  or  they  are  not  reporting 
to  us  those  cases  in  which  they  were  engaging 
in  collusion  or  other  price-fixing  methods. 
Great  tact  and  care  are  necessary  at  this 
point.  It  is  important  to  make  the  contrac¬ 
tor  understand  that  we  do  not  want  those  cases 
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which  are  not  truly  competitive  bidding 
situations,  but  we  do  want  those  cases  that 
are  truly  competitive  bidding  situations, 
regardless  of  how  foolish  the  contractor 
might  have  been  at  the  time  of  bidding.  Thus, 
we  not  only  establish  a  baseline  for  reference 
at  a  future  data,  after  changing  methods  of 
bidding,  but  we  also  use  baseline  data  to  help 
us  determine  whether  or  not  the  dataset  we  are 
working  with  is  actually  the  dataset  we  think 
we  are  working  with. 


STEP  3:  MODELING 


At  least  six  sub-steps  are  necessary  in 
modeling  competitive  bidding  problems: 

(1)  identify  controllable  variables;  (2)  iden¬ 
tify  uncontrollable  variables;  (3)  develop 
a  long-term  mathematical  model;  (4)  find  a 
decision  rule  to  achieve  the  objectives; 

(5)  modify  the  rule  as  is  necessary  to  fit 
temporary  or  permanent  constraints;  and 

(6)  modify,  if  necessary,  to  better  achieve 
short-term  objectives. 

The  controllable  variables  are  identified 
through  our  own  knowledge  of  construction 
operations  and  interviews  with  concerned  par¬ 
ties.  Examples  of  controllable  variables 
are  crew  composition  in  the  case  of  studies 
of  productivity,  rate  of  hire  in  the  case  of 
studies  of  job  completion  rates,  and  level  of 
detail  in  cost  estimates  in  the  case  of  com¬ 
petitive  bidding  strategies. 

Identification  of  uncontrollable  variables 
also  depends  primarily  upon  prior  experience 
in  the  area,  and  interviews  with  persons 
intimately  involved  in  the  construction 
operation.  Some  of  the  uncontrollable  vari¬ 
ables  include  the  number  and  types  of  com¬ 
petitors  who  will  be  submitting  bids,  the 
size  of  the  project,  short-term  cyclic  ef¬ 
fects,  seasonal  variations  of  the  market, 
and  long-term  (say,  five  or  six  year)  cyclic 
effects  in  the  marketplace. 

By  identifying  the  major  variables,  we  de¬ 
velop  mathematical  models  involving  both 
parametric  and  random  variable  terns.  We 
find  that  it  is  necessary  to  explain  to  the 
client  the  importance  of  the  form  of  each 
parametric  term  as  well  as  the  method  of  esti¬ 
mating  parameters  in  the  model.  We  have 
found  the  thesis  by  Ludolph  (10)  to  be  use¬ 
ful  in  explaining  our  approach  to  contractors. 
We  have  also  found  that  write-ups,  such  as 
those  in  Appendix  B  to  this  paper  which  was 
taken  from  the  thesis  by  Li  (8) ,  are  also 
useful  in  explaining  to  a  contractor  how  we 
find  an  optimum  pricing  policy. 


Part  of  the  modeling  problem  is  to  find 
an  approximation  for  random  variable  terms, 
including  especially  the  unexplained  vari¬ 
ability.  In  the  case  of  competitive  bid¬ 
ding,  this  so-called  unexplained  variability 
is  the  prediction  error  as  perceived  by  a 
subject  contractor.  In  other  words,  it 
is  the  subject's  perception  of  the  market 
varying  around  his  estimate  of  the  cost  to 
construct  a  project.  We  generally  use 
the  RS  distribution  to  represent  unimodal 
random  variables.  Appendix  A  from  Li  (8), 
which  is  a  rewrite  of  sections  from  pre¬ 
vious  theses  by  Fantozzi  (2)  and  Grieve 
(5) ,  explains  how  to  use  the  first  four 
sample  moments  to  estimate  parameters. 

After  modeling  the  problem  at  hand,  we  find 
optimum  solutions.  In  the  case  of  com¬ 
petitive  bidding,  this  means  that  we  find 
a  markup  policy  that  will  best  achieve  the 
contractor's  objectives.  It  is  important 
to  note  that  the  decision  rules  developed 
are  long-term  rules;  that  is,  they  are 
the  rules  that,  in  the  long  run,  will  best 
achieve  management's  objectives. 

After  developing  long-term  rules  for  stan¬ 
dard  unconstrained  conditions,  we  examine 
the  operation  to  determine  if  there  are 
constraints  that  would  cause  us  to  modi¬ 
fy  the  rule.  If  necessary,  rules  are 
modified  to  fit  existing  constraints.  In 
the  competitive  bidding  environment,  the 
problem  is  that  many  companies  are  con¬ 
strained  either  by  the  number  of  bids  they 
may  submit  in  a  given  time  period,  or  by 
the  number  of  jobs  they  may  submit  in  a 
given  time  period. 

This  is  illustrated  in  Figure  1,  taken 
from  Li  (8) .  This  figure  shows  equal 
profits  contour  lines  for  a  construction 
company.  Profits  are  maximized  if  the 
firm  is  able  to  complete  j**  projects  in 
a  time  period,  and  is  able  to  bid  b** 
projects  in  a  time  period.  However,  if 
the  firm  is  unable  to  bid  or  complete  the 
required  number  of  jobs  to  maximize  pro¬ 
fits,  then  another  bidding  policy  must  be 
adopted,  depending  upon  the  constraints 
that  exist  at  the  time  of  bidding. 

It  is  noted  that  it  is  sometimes  not  pos¬ 
sible  to  analytically  model  complex  bid¬ 
ding  environments.  We  often  turn  to  com¬ 
puter  simulation.  An  example  of  such  a 
study  of  constraints  is  a  thesis  by  Rhye 
(13)  and  his  research  concerning  the  in¬ 
fluence  of  the  backlog  of  work  on  construc¬ 
tion  operations.  (Workshop  participants 
who  may  be  interested  in  the  simulation 
approach  to  examination  of  risk  under  con¬ 
strained  conditions  might  wish  to  contact 
Captain  Rhye,  an  Air  Force  officer  cur¬ 
rently  stationed  at  the  Air  Force  Academy) . 


Ill 


j 


b 

Figure  1.  The  Four  Domains  (From  Reference  8) 
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Finally,  after  developing  a  long-term  policy 
and  modifying  that  policy  as  necessary  when 
constraints  do  exist,  we  test  the  policy  to 
insure  that  in  striving  to  achieve  long-term 


we  are  also  doing  are  best  in  the  short  run 
An  example  from  Li  (8)  is  shown  in  Figure  2 
This  study  involved  the  use  of  the  binomial 
distribution  to  transform  long-term  bidding 
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probabilities  into  short-term  bidding  probabili¬ 
ties,  and  to  evaluate  the  profits  that  might  be 
realized  when  only  a  small  number  of  projects 
are  being  bid  in  a  time  interval.  From  the 
study  illustrated  in  Figure  2,  it  was  found  that 
by  striving  to  achieve  long-term  objectives,  the 
contractor  does  indeed  achieve  short-term  objec¬ 
tives  as  well.  That  is,  long-term  pricing  poli¬ 
cies  that  are  best  in  the  long  run  are  also  best 
in  the  short  run. 

It  has  been  our  experience  that  on  first  glance, 
nearly  all  problems  encountered  in  the  construc¬ 
tion  industry  are  'bizarre”  in  the  sense  that 
they  are  poorly  behaved,  multimodal  distribu¬ 
tions.  In  all  cases  studied  to  date,  when  a 
relatively  few  number  of  independent  variables 
(quantitative  and/or  qualitative)  are  properly 
included  in  our  mathematical  models,  the  unex¬ 
plained  variability  is  greatly  reduced  and  it 
becomes  unimodal,  and  falls  within  the  range  of 
skewness  and  kurtosis  tables  (which  are  includ¬ 
ed  in  Appendix  C  to  this  paper).  Distributions 
outside  the  range  of  the  tabled  values  have  in¬ 
variably  been  found  to  be  reducable  by  the  in¬ 
troduction  of  an  additional  explanatory  vari* 
able . 

It  should  also  be  noted  that  in  our  experience, 
the  unexplained  variability  (that  is,  the  pre¬ 
diction  error)  is  usually  significantly  skewed 
and  more  peaked  than  a  normal  distribution.  We 
seldom  find  any  justification  for  the  use  of 
uniform,  triangular,  trapezoidal,  Weibull  or 
PERT-type  distributions.  All  of  these  distri¬ 
butions  are  typically  much  too  symmetrical  and 
too  flat  to  represent  real  life  data.  The  only 
justification  we  have  found  for  using  such  dis¬ 
tributions  is  that  they  are  convenient  for  sim¬ 
plistic  practitioner  applications.  Given  the 
fact  that  we  have  complex  problems  with  signi¬ 
ficant  impacts  both  in  government  and  industry, 
clearly  the  time  has  come  to  use  appropriate 
distributions  that  best  represent  reality,  as 
distinguished  from  those  that  are  convenient 
for  the  analyst. 

We  have  also  found,  by  experience,  that  the  RS 
distribution  is  by  far  superior  to  all  other 
percentile-type  distributions  for  approximating 
unimodal  random  variables.  We  insist  upon  a 
percentile  form  of  distribution  since  we  must 
often  turn  to  simulation,  and  this  form  is  best 
suited  for  simulation  applications.  The  RS  dis¬ 
tribution,  its  use  in  our  pricing  studies,  and 
values  of  its  lambda  parameters  are  attached 
to  this  paper.  The  lambda  values  from  Ricer's 
thesis  (14),  Appendix  C,  are  found  after  calcu¬ 
lating  the  sample  moments.  However,  data  is  of¬ 
ten  unavailable  for  many  types  of  studies  and 
we  therefore  often  use  nonlinear  regression  to 
fit  conceptual  estimates  of  the  distribution 
function  of  the  RS  distribution.  Alternately, 
when  we  are  working  in  areas  in  which  we  have 
considerable  prior  knowledge  of  the  density 
functions  being  encountered,  we  may  directly 
estimate  the  first  two  moments  and  the  shape 
(skewness,  kurtosis)  of  the  random  variable  at 


hand.  Our  choice  of  the  method  of  estimating 
parameters  depends  entirely  upon  the  data  avail¬ 
able  and  our  experience  in  studying  similar 
types  of  random  variables. 

Four  major  problem  areas  are  sometimes  encoun¬ 
tered  in  our  studies: 

1.  Nonlinear  regression  programs,  such  as  BMDP- 
3R,  are  unable  to  handle  models  containing 
two  or  more  nonlinear  independent  variables. 
To  overcome  this  problem,  we  often  work  with 
successive  generations  of  residuals. 

2  Practical  continuous  simulation  programs  are 
not  able  to  handle  correlated  distributions. 
We  have  developed  some  of  our  own  programs 
to  overcome  this  barrier.  The  first  such 
program  was  developed  by  Frost  (3)  for  stu¬ 
dies  of  competitive  bidding.  This  program 
has,  however,  been  used  to  study  all  types 
of  problems  when  random  variables  are  cor¬ 
related  with  one  another.  A  very  clever  al¬ 
gorithm  was  recently  developed  by  Lin  (9) 
while  developing  a  general  model  that  links 
together  all  known  theories  of  the  probabi¬ 
lity  of  winning  in  a  competitive  bidding  en¬ 
vironment  . 

3.  Missing  data  in  regression  studies  is  ano¬ 
ther  major  problem  we  often  encounter.  To 
understand  this  problem,  suppose  one  has 
1000  projects  and  wishes  to  estimate  the 
parameters  for  200  variables  that  are  en¬ 
countered  in  these  projects.  Assume  that 

a  linear  multivariate  model  is  appropriate. 
Also  suppose  that  any  one  project  includes 
only  30  to  50%  of  the  variables.  Under 
such  conditions,  when  we  attempt  to  run  a 
simple  linear  regression  program,  all  pro¬ 
jects  with  missing  data  are  thrown  out.  In 
fact,  we  end  up  with  no  projects  remaining 
in  our  dataset,  and  we  are  unable  to  esti¬ 
mate  the  values  of  the  parameters. 

4.  Network  representation  of  projects  has  also 
posed  problems.  Based  on  a  number  of  at¬ 
tempts  to  predict  the  time  to  complete  pro¬ 
jects  using  either  arrow  or  precedence-type 
diagrams,  the  author  has  concluded  that 
while  these  are  excellent  tools  for  planning 
projects  and  communicating  interrelation¬ 
ships,  networks  are  totally  unacceptable 
for  predicting  and  evaluating  the  time  re¬ 
quired  for  completing  projects.  The  work 

by  Graf  (2)  explains  in  some  detail  why 
these  models  are  inappropriate  for  time  pre¬ 
dictions.  He  suggested  that  we  must  find 
and  use  other  methods  for  modeling  if  our 
objective  is  to  predict  time  required  on  a 
project.  We  are  currently  attempting  to 
develop  some  new  approaches  in  this  area. 

Another  problem  of  growing  concern  is  the  need 
to  develop  a  test  for  Lack  of  Fit.  Ludolph  (10) 
has  addressed  this  problem  using  the  WSSD  sta¬ 
tistic.  However,  we’re  not  yet  satisfied  with 
our  approach  to  the  Lack  of  Fit  problem.  In 
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construction,  our  major  problem  is  that  the  as¬ 
sumption  of  independence  among  all  observations 
is,  of  course,  false.  This  means  that  any  time 
we  attempt  to  run  a  Lack  of  Fit  test  on  a  selec¬ 
ted  model,  it  fails  for  lack  of  fit.  Since  all 
models  fail  due  to  this  problem,  we’re  not  in¬ 
clined  to  perform  Lack  of  Fit  tests.  While 
this  is  not  presently  a  top  priority  problem 
for  us ,  it  is  one  that  the  author  feels  should 
be  addressed  in  the  very  near  future.  (Note: 
we  have  no  problem  handling  missing  data  in  ty¬ 
pical  time  series  problems.  Our  problem  is  in 
other  domains,  such  as  size,  quantity  and  cost, 
when  large  proportions  ranging  from  10  to  80% 
of  the  cases  are  missing  in  each  of  these  do¬ 
mains  . 

STEP  4;  APPLYING 

Before  applying  new  decision-making  rules  in 
any  operation,  we  suggest  that: 

1.  A  cum  sum  control  chart  should  be  developed 
so  that  sudden  changes  In  the  enterprise  or 
environment  will  be  promptly  detected  (we 
call  this  a  change  in  modus  operandi) . 

2.  The  expected  results  should  be  reviewed  one 
more  time  before  the  model  is  put  into  use. 

3.  A  number  of  cases  or  an  elapsed  time  period 
before  passing  any  judgments  or  evaluating 
the  results  should  be  re-approved  by  the 
chief  executive  officer. 

The  greatest  problem,  In  the  author’s  exper¬ 
ience,  when  applying  any  new  approach  is  that 
persons  working  on  one  project  want  to  judge 
new  decision-making  rules  by  the  outcome  of  the 
next  one  in  which  he  has  an  interest.  As  a 
general  rule,  in  the  case  of  competitive  bid¬ 
ding,  If  we  happen  to  win  the  first  three  jobs 
using  the  new  policy  management  will  immediate¬ 
ly  fully  endorse  the  policy  even  if  we  only  ex¬ 
pected  to  win  1  In  20.  Conversely,  if  we  don't 
win  the  first  4  or  5  jobs  using  a  new  bidding 
policy,  the  entire  project  is  likely  to  be  re¬ 
jected  even  though  the  bidding  policy  might  be 
designed  to  win  80%  of  the  jobs  in  the  long  run. 
For  these  reasons,  it  is  absolutely  essential 
that  there  be  a  pre-agreement  on  the  number  of 
projects  to  be  bid  before  any  judgments  are 
passed . 

STEP  5 :  EVALUATING 

We  have  found  that  new  pricing  policies  will 
double  the  profits  of  most  companies.  As  a 
general  rule,  contractors  are  able  to  secure 
50%  of  the  so-called  ’’dollars  in  the  market” 
using  analytical  methods.  Comparing  these  fig¬ 
ures  to  those  cited  in  the  Baselining  Section 
(Step  2) ,  it  is  noted  that  this  means  that  pro¬ 
fits  are  typically  doubled  by  turning  from  pre¬ 
sent  methods  to  new  analytical  approaches  in 
competitive  bidding. 


We  have  also  found  another  rule  of  thumb:  In 
any  six-month  period  of  time,  the  profits  ac¬ 
tually  obtained  by  an  analytical  approach  would 
be  about  90%  of  those  that  were  predicted,  i.e., 
there  is  a  shrinkage  of  about  10%  in  realiza¬ 
tion  of  profits  as  compared  to  the  predicted 
amounts .  This  is  due  to  a  combination  of  sam¬ 
pling  errors,  changing  markets,  estimating  er¬ 
rors,  and  other  factors.  Despite  this  shrink¬ 
age,  the  average  firm  can  increase  its  profits 
by  about  1%  on  sales. 

WTien  evaluating  the  results  of  competitive  bid¬ 
ding  policies,  we  often  find  that  the  client 
did  not  consistently  use  the  proposed  model 
throughout  the  trial  period.  However,  as  should 
be  the  case,  it  is  found  that  in  the  long  run, 
his  attempts  to  beat  the  analytical  model  are 
to  no  avail.  However,  we  carefully  review  those 
instances  in  which  the  subject  contractor  did 
feel  that  a  special  condition  was  being  encoun¬ 
tered  and  we  consider  this  condition  in  future 
modeling  to  determine  whether  it  does  indeed  im¬ 
pact  the  bidding  model. 

GENERAL  RECOMMENDATIONS  TO  DOD 

Based  on  the  author’s  experiences  in  studies  of 
competitive  bidding  and  other  risk  and  uncer¬ 
tainty  problems  in  the  construction  industry, 
he  offers  five  suggestions  to  DCD  for  improving 
its  studies  of  risk  and  uncertainty  in  large 
projects : 

1.  There  is  a  need  for  a  trained  core  of  spe¬ 
cialists  who  are  regularly  engaged  in  the 
analysis  of  data  from  several  sources. 
Diagnostic  techniques  and  modeling  skills 
cannot  be  maintained  by  persons  working  on 
one  project  or  within  one  agency  when  they 
are  looking  at  very  few  datasets.  To  main¬ 
tain  the  necessary  skills,  one  must  continue 
to  view  and  explore  a  wide  variety  of  data. 
We  find  that  it  takes  between  15  and  18 
months  for  a  Master  of  Science  graduate 
student  to  become  competent  and  comfortable 
In  the  application  of  modern  techniques  for 
modeling  and  assessing  management  and  uncer¬ 
tainty. 

2.  Individuals  responsible  for  performing  ma¬ 
jor  risk  and  uncertainty  problems  should 
serve  as  consultants  to  the  responsible 
agency  and  should  not  be  subordinates  of 
the  responsible  manager.  Time  and  again, 
we've  seen  competent  study  results  altered 
to  yield  policies  that  best  fit  the  self- 
interests  of  superiors.  This  is  sometimes 
intentional,  sometimes  unintentional.  In 
any  event,  it  happens  often  enough  to  justi¬ 
fy  a  consultancy  relationship  when  perform¬ 
ing  major  studies  of  risk  and  uncertainty. 

3.  DOD  should  cooperate  with  other  federal 
agencies  and  the  private  sector  to  establish 
a  national  data  bank  for  future  studies  of 
risk  and  uncertainty.  The  author  has  been 
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urging  the  creation  of  such  a  data  bank  for 
a  number  of  years.  The  rationale  behind 
this  reconmendation  is  that  the  data  must 
be  collected  and  stored  as  projects  pro¬ 
gress;  otherwise,  much  data  that  will  be 
needed  in  future  analyses  will  never  be  col¬ 
lected  and  properly  recorded.  If  such  a 
data  bank  is  not  established,  what  little 
information  that  is  collected  which  might 
be  of  use  to  us  is  eventually  sent  to  the 
National  Archives  for  aging  ...  anyone  who 
has  ever  tried  to  recover  information  from 
the  National  Archives  would  agree  that  the 
time  and  cost  of  retrieval  are  prohibitive. 

4.  The  Department  of  Defense  should  establish 
a'  large  number  of  graduate  student  intern¬ 
ships  for  M.S.  and  Ph.D.  students,  so  that 
thesis  and  dissertation  research  may  be  di¬ 
rected  toward  DOD  risk  and  uncertainly 
problems.  However,  research  should  not  be 
limited  to  those  attending  military  schools. 
The  fact  is,  DOD's  uncertainty  and  risk 
problems  are  not  unique  to  that  agency  and 
graduate  student  research  geared  toward 
solving  problems  in  other  agencies  and  in 
the  private  sector  will  most  assuredly 
benefit  DOD  as  well. 

5.  The  Department  of  Defense  should  use  its 
influence  to  address  the  basic  and  criti¬ 
cal  issue  of  the  need  for  significant  finan¬ 
cial  support  for  software,  hardware  and  ad¬ 
ditional  faculty  in  universities  and  col¬ 
leges  where  future  managers  are  now  being 
trained.  Given  the  present  trend,  we  are 
going  to  have  very  few  DOMESTIC  M.S.  and 
Ph.D.  graduates  between  now  and  the  year 
2000.  National  statistics  Indicate  that 
nearly  half  of  all  graduate  degrees  are 
being  awarded  to  international  students , 
many  of  whom  are  fully  supported  financial¬ 
ly  by  their  governments  and  native  busines¬ 
ses.  These  numbers  are  increasing  every 
year.  Given  the  current  trend,  we  can  ful¬ 
ly  expect  that  by  the  end  of  this  century, 
the  majority  of  real  experts  in  the  field 

of  uncertainty  and  risk  will  be  expatriots 
residing  in  their  native  countries,  solv¬ 
ing  the  problems  of  their  governmental  agen¬ 
cies  and  industries  and  establishing  their 
own  educational  programs  that  will  inevi¬ 
tably  far  surpass  our  own.  If  this  trend 
continues  unabated,  we  will  be  in  the  un¬ 
enviable  position  of  having  to  reimport  our 
own  technology  because  we  will  have  become 
incapable  of  training  experts  of  our  own. 

CONCLUSIONS 


In  this  paper,  the  author  has  discussed  some 
practical  applications  of  risk  and  uncertainty 
analysis  In  construction  operations.  In  parti¬ 
cular,  he  has  reported  applications  and  dis¬ 
cussed  successes  and  failures  relative  to  com¬ 


petitive  bidding  strategies.  The  approach  re¬ 
ported  in  this  paper  has  been  applied  from  Che 
buyer* s  side  as  well  as  the  seller*s  side  in 
other  studies.  The  approach  has  been  applied 
in  other  areas,  such  as  that  of  quality  con¬ 
trol.  It  has  been  applied  In  the  design  of 
labor-intensive  crews  and  in  capital-invensive 
spreads.  It  is  the  author's  opinion  that  the 
methods  reported  are  transferable  to  DOD  large 
projects . 

Perhaps  the  major  problem  In  transferring 
these  methods  to  DOD  Is  the  same  in  trans¬ 
ferring  the  methods  into  the  construction  in¬ 
dustry.  Responsible  persons  seem,  almost  uni¬ 
versally,  to  have  little  If  any  training  In 
statistical  methods,  the  use  of  computers,  and 
basic  concepts  of  risk  and  uncertainty.  Per¬ 
sons  responsible  for  analysis  are  not  speci¬ 
fically  trained  In  the  methods  required  in 
modern-day  studies.  They  think  like  engineers 
or  they  think  like  mathematicians.  Unfortu¬ 
nately,  they  have  great  difficulty  thinking  in 
terms  of  random  variables  and  thinking  in 
terms  of  the  interactions  and  transformations 
that  occur  among  and  through  random  variables. 
The  author  has  offered  no  suggestion  as  to  how 
this  problem  in  thinking  and  understanding  of 
responsible  persons  can  or  should  be  addressed 

However,  the  author  has  suggested  that  we  need 
to  begin  to  look  ahead  to  the  next  20  years , 
and  to  train  a  cadre  of  experts  who  can  serve 
as  consultants  to  responsible  persons.  The 
author  has  suggested  that  we  need  to  imple¬ 
ment  internship  programs ,  to  increase  sup¬ 
port  for  those  who  are  providing  training  for 
future  managers  of .major  acquisition  activi¬ 
ties,  and  the  author  has  noted  that  unless 
changes  are  made,  this  nation  will  not  only 
be  dependent  upon  foreign  governments  for 
oil  and  other  resources,  but  will  be  depen¬ 
dent  upon  foreign  governments  for  Its  supply 
of  persons  expert  in  the  field  of  uncertainty 
and  risk. 
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APPENDIX  A  —  THE  R-S  DISTRIBUTION* 


The  residuals  associated  with  the  predicted 
equation  represent  the  unexplained  variation 
corresponding  to  the  predicted  equation's  in¬ 
ability  to  properly  describe  the  observed  mark¬ 
ups.  One  can  account  for  this  variability  by 
describing  the  distribution  of  the  residuals. 
The  R-S  distribution  is  useful  for  the  repre¬ 
sentation  of  data  when  the  underlying  model  is 
unknown,  since  a  wide  variety  of  curve  shapes 
is  available  with  this  distribution. 

The  R-S  distribution  is  a  generalization  of 
Tukey's  lambda  (percentile)  distribution.  A 
percentile  distribution  characterizes  a  random 
variable  as  a  function  of  its  cumulative  prob¬ 
ability  (from  0  to  1).  This  works  well  in 
pricing  studies  since  X  (markup)  is  a  function 
of  P  (probability  of  winning) .  The  standard 
form  of  the  function  is: 


X  =  R(P)  =  Xx  +  (PA3  -  (1-P)A4)/X2, 


(0  <  P  <  X) 


(A.  1) 


where  X 


P 


X2 

*3>*4 


the  standardized  value  of  the  ran¬ 
dom  variable; 

the  lOO-pth  percentile  at  which  the 
function  is  being  evaluated; 
the  location  parameter; 
the  scaling  parameter;  and, 
the  shape  parameters. 


The  expected  value  of  X  will  be: 


E(X) 


/  R(P)  dP  =  X  + 
p-0  3 


1 

X.  4-1 
4 


9A2 


(A. 2) 


The  density  function  of  any  percentile  distri¬ 
bution  is  the  inverse  of  the  derivative  of  the 
distribution  function: 


f(X)  «  f (R(P) )  -  ^  R(P)"1 


(A. 3) 


X3-l  a4-i 

Xz/CX3  P  +  Vx-P)  )  (A‘4) 


The  term  R(P)  in  Equations  A.l  and  A. 2  desig¬ 
nates  the  standardized  R-S  Distribution  whose 
mean  (y)  is  zero  and  variance  (a^)  is  one 
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*Taken  From  Li  (8) ,  which  was  derived  from 
Fantozzi  (2),  Grieve  (5),  Larew  (7),  and 
Dudewicz  (1). 


(A.  14) 


Tables  of  \  through  4  can  be  found  in  works 
by  Dudewicz  (1),  Ramberg  (11,  12),  Tadikamalla 
(15),  Larew  (7),  and  Ricer  (14),  and  are  in¬ 
cluded  as  Appendix  C  to  this  paper. 


—  4 

m4  =  Z  (X  -  X)  /n 
i*l 


The  R-S  distribution  can  also  be  used  to  ap¬ 
proximate  actual  data.  In  this  case,  the  fit¬ 
ted  form  of  the  equation  becomes  necessary: 


2.  Calculate  the  standardized  (dimensionless) 
third  and  fourth  moments  of  the  sample: 


X  = 

R(P)  =  y  +  oR(P) 

(A. 5) 

Y3  = 

,  3/2 

m3/m2 

(A. 15) 

E(X)  = 

y  +  oe (R(P)) 

(A.  6) 

Y4  * 

.  2 

Vm2 

(A. 16) 

f(X)  = 

f(R(P))/o 

(A. 7) 

3.  Match 

the  values  of  | “V 3 1 

and  | Y4 |  with  the 

nearest  table  values  of  a  3  and  04,  and  re- 

,  .  ,  cord  the  values  of  A  A  A.  and  A., 

where  y  is  the  population  mean  and  a  is  the  123  4 

population  variance. 


The  mean  and  variance  can  be  estimated  by  the 
first  (mi)  and  second  (m2)  sample  moments. 


4.  If  Y3  is  negative,  interchange  the  values 
of  and  a^,  and  change  the  sign  of  A^. 


The  fitted  values  of  the  distribution  func¬ 
tion,  the  expected  value  and  the  density 
function  will  become: 

X  *  R(P)  =  +  /S^  (Xx  +  P  3  -  (1-P)X4)A2 

(A.  8) 

E(X)  -  +  /5^(X1  +  ((X3+I)'1  -  (X4+U-1)/X2) 


The  R-S  distribution  can  also  be  used  to  ap- 
proximade  many  of  the  widely  used  familiers 
of  continuous  uniraodal  distributions.  Fig¬ 
ure  3  shows  some  common  unimodal  distribu¬ 
tions  and  their  locations  on  the  skewness- 
kurtosis  plane.  Studies  at  The  Ohio  State 
University  have  shown  that  the  R-S  distribu¬ 
tion  gives  an  excellent  approximation  of  the 
random  variables  commonly  found  in  construc¬ 
tion  costing  and  pricing  studies. 


(A.  9) 

~  _  A3-I  A^-i 

f(X)  =  X2/(vm2(X3P  J  +  X4(l-P)  4  )  (A. 10) 

To  choose  the  four  lambda  parameters,  the 
skewness  (the  third  standardized  moment 
which  is  a  measure  of  the  symmetry  of  the 
probability  density  function)  and  kurtosis 
(the  fourth  standardized  moment  which  is 
a  measure  of  the  peakedness  of  the  probab¬ 
ility  density  function)  of  the  random  vari¬ 
able  must  be  estimated.  This  is  accom¬ 
plished  by  the  following  procedure: 


Calculate  the  first,  second, 
fourth  sample  moments: 

third  and 

n 

m.  »  X  *  Z  X  /n 

i=l  1 

(A. 11) 

n  —  2 

nu  =  Z  (X  -  X)  /n 
i*l 

(A. 12) 

m 


3 


I  (X  -  X) 3/n 
i=l 


(A. 13) 


Distributions  (5). 
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APPENDIX  B  —  M-STAR  METHODOLOGY* 


Pricing  studies  performed  by  Larew  (7)  indi¬ 
cate  that:  (1)  markups  of  competitors  may  be 
expressed  as  a  function,  of  project  size; 

(2)  markups  may  or  may  not  be  independent  of 
project  size;  and  (3)  economies  or  disecono¬ 
mies  of  scale  may  exist  in  a  competitively  bid 
market.  Based  on  these  findings,  Larew  devel¬ 
oped  an  equation  for  predicting  the  response 
variable,  markup,  as  a  function  of  the  inde¬ 
pendent  variable,  estimated  cost  (or  estimated 
project  size).  The  equation  is: 

M  =  A  +  CXk  (B.l) 

where 

M  -  the  markup  as  a  percentage  or  pro¬ 
portion  of  the  estimated  cost; 

X  -  the  estimated  cost; 

A  -  an  estimate  of  the  constant  percent¬ 
age  added  to  any  project  irrespec¬ 
tive  of  project  size; 

C  =  an  estimate  of  the  constant  of  pro¬ 
portionality;  and, 

k  =  an  estimate  of  the  economy  of  scale 
coefficient . 


Larew  found  that  markups  tend  to  decrease  as 
project  size  increases,  with  the  economy  of 
scale  coefficient  generally  ranging  from  0  to 
-1.  These  preliminary  findings  motivated  the 
development  of  the  M*  bidding  strategy 

Using  the  above  relationship,  one  begins  to 
formulate  a  bidding  policy  by  fitting  the  ob¬ 
served  perceived  markups  of  the  low  bidder  as 
a  function  of  estimated  project  cost  for  all 
past  projects  in  a  given  market  or  class  of 
work.  (Obviously,  the  contractor  with  no  in¬ 
formation  of  past  competitively  bid  projects 
need  not  consider  this  analytical  procedure 
and  must  continue  conceptual  pricing  practices 
until  a  data  base  is  established)  .  The  per¬ 
ceived  markup  is  found  by  the  relationship: 


contractor,  the  perceived  markup  represents 
the  actual  markup  applied  to  the  cost  esti¬ 
mate.  For  projects  won  by  a  competitor,  the 
perceived  markup  is  a  perception  of  the  com¬ 
petitor’s  pricing  policy  with  respect  to  the 
contractor's  estimated  cost.  Figure  4  shows 
a  plot  of  perceived  markups  versus  estimated^ 
project  costs  and  the  fitted  line,  M  **  A  +  CX  . 
The  relationship  between  the  two  variables, 

M  and  X,  is  statistical  and  residuals  (or  er¬ 
rors)  may  be  associated  with  each  observation 
with  respect  to  the  fitted  line.  Residual  is 
defined  as  the  observed  markup  minus  the  pre¬ 
dicted  markup;  thus, 

Residual  =  (Observed  M)  -  (A  +  CXk) 

(B  .3) 

The  residuals  represent  some  unexplained  var¬ 
iability  in  the  observations  and  may  be  ap¬ 
proximated  by  the  R-S  distribution  if  homo- 
scedasticitv  of  the  residuals  is  obtained. 
Homoscedast icity  exists  if  the  mean  of  the 
residuals  is  zero  and  the  variance  around 
the  fitted  equation  is  constant  over  the  en¬ 
tire  range  of  the  independent  variable. 

Figure  5  shows  a  residual  plot  where  the  re¬ 
siduals  may  be  considered  homoscedastic.  The 
zero  residual  line  in  this  plot  represents 
the  value  of  the  markup  found  by  the  fitted 
equation,  M  =  A  +  CX^.  It  is  often  difficult 
to  visually  test  for  homoscedasticity  since 
datasets  are  relatively  small;  however,  one 
must  look  for  trends  in  the  residual  plot  to 
make  the  assumption  that  homoscedasticity 
does  or  does  not  exist.  Figure  6  shows  a 
residual  plot  where  homoscedasticity  does  not 
exist.  The  absence  of  homoscedasticity  re¬ 
quires  further  refinement  of  data  and/or 
additional  analysis  to  remove  some  unex¬ 
plained  quantitative  or  qualitative  factor. 

Assuming  homoscedasticity  exists,  the  predic¬ 
tion  equation  is  improved  by  including  a  de¬ 
scription  of  the  residuals,  R(P) ,  approxi¬ 
mated  using  the  R-S  distribution,  and  the 
equation  becomes: 

M  =  A  +  CXk  +  R(P)  (B.4) 


Perceived  Low  Bid  -  Subject  Cost  Estimate 
Markup  Subject  Cost  Estimate 

(B .  2) 


All  projects,  won  or  lost,  should  be  included 
in  the  data  base.  For  projects  won  by  the 

*This  appendix  is  taken  from  Li  (8) .  The  for¬ 
mat  and  some  comments  in  this  section  were  ta¬ 
ken  from  Fantozzi  (2)  and  reproduced  in  Appen¬ 
dix  A  of  Rhye’s  thesis  (21). 


The  R-S  distribution  is  a  percentile  distri¬ 
bution  that  characterizes  a  random  variable 
as  a  function  of  its  cumulative  probability. 

As  explained  in  Appendix  A,  to  use  the  R-S 
distribution,  one  must  first  determine  the 
first,  second,  third  and  fourth  sample  mo¬ 
ments  of  the  residuals  around  the  fitted  line, 
A  +  CXk,  and  then  standardize  (i.e.,  make 
dimensionless)  the  third  and  fourth  moments 
which  are  measures  of  the  symmetry  (skewness) 
and  peakedness  (kurtosis),  respectively,  of 
the  residuals  around  the  fitted  line.  The 
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Figure  4.  Perceived  Markup  versus  Estimated  Project  Size  (After  Rhye  (13)) 
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Figure  5.  Homoscedas tic  Residuals  (After  Rhye  (13)) 
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Figure  6.  Non-Homos cedas tic  Residuals  (After  Rhye  (13)) 
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impact  of  these  calculations  may  be  understood 
by  examining  the  probability  distribution  func¬ 
tion  (pdf)  of  the  residuals.  Construction  of 
the  pdf  may  be  visualized  bv  rotating  the  resi¬ 
dual  plot,  such  as  that  shown  in  Figure  5,  90 
degrees  clockwise  and  mapping  the  residuals 
down  to  the  residual  axis.  One  may  visualize 
the  construction  of  a  histogram  for  the  resi¬ 
duals,  shown  in  Figure  7,  such  that  each  resi¬ 
dual  is  mapped  into  the  appropriate  interval. 
The  dotted  line  in  Figure  7  represents  the  pdf 
of  the  residuals.  This  distribution  appears  to 
be  negatively  skewed;  thus,  the  third  standard¬ 
ized  moment  will  be  some  value  less  than  zero 
(the  skewness  of  a  symmetrical  distribution, 
such  as  the  normal  distribution,  is  zero).  It 


is  very  difficult  to  visually  estimate  the 
peakedness  of  a  distribution;  however,  this 
distribution  appears  to  be  more  peaked  (a  high¬ 
er  kurtosis  value)  than,  for  example,  a  normal 
distribution . 

While  the  pdf  may  be  constructed  with  the  R-S 
distribution  by  taking  the  inverse  of  the  deri¬ 
vative  of  R(P) ,  the  cumulative  distribution 
function  is  developed  using  the  R-S  distribu¬ 
tion  and  the  above  calculated  moments,  since 
markup  is  a  function  of  the  probability  of  win¬ 
ning.  The  cdf  is  constructed  by  iteratively 
determining  the  expected  value  of  the  residual 
(markup)  for  various  probabilities  of  occur¬ 
rence  (from  0  to  1),  as  shown  in  Figure  8. 


Figure  7.  Histogram  of  Residuals  (After  Li  (8)). 


Figure  8.  Cumulative  Density  Function:  Markup  as  a  Function  of  P  (After  Li  (8)). 
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APPENDIX  C  —  LAMBDA  PARAIffiTERS 


The  probability,  P,  may  be  interpreted  as  the 
probability  of  not  winning  a  contract  at  the 
corresponding  markup;  therefore,  the  prob¬ 
ability  of  winning  is  (1  -  P) . 

The  above  information  may  now  be  used  to  de¬ 
velop  a  bidding  strategy  aimed  at  maximizing 
expected  net  profits.  Net  profits  for  a  pro¬ 
ject  may  be  considered  to  equal  the  markup 
minus  the  costs  of  overhead  and  estimating, 
and  expectancy  theory  states  that  the  expect¬ 
ed  net  profits  for  a  project  are: 


E(n)  -  (A  +  CXk  +  R(P))(1-P)  -  C  (1-P) 


-  C 


es  t 


where 


(B.  5) 


k  E(H) 
A  +  CX  +  R(P) 

(1  -  P) 


estimated  net  profits; 
markup ; 

probability  of  winning  at 
the  above  markup; 
cost  of  overhead  function; 
and , 

cost  of  estimating  func¬ 
tion. 


The  following  tables  are  taken  from  Appendix 
H  of  Ricer  (14).  Values  are  those  currently 
used  at  The  Ohio  State  University.  They  have 
been  selected  from  tables  previously  published 
by  Tadikamall a,  Ramberg,  Sclimei.se r,  Dudewicz 
and  Mykvtka  (1,  11,  12,  15) . 


The  expected  net  profits  are  maximum  for  the 
above  relationship  when  E (II)  =  0.  Taking 
the  derivative  of  the  above  equation  and  set¬ 
ting  it  equal  to  zero  gives: 

(l-P)  (R(P) )  =  A  +  CXk  +  R(P)  -  CQ 

(B.6) 


The  above  equation  may  be  solved  by  iterating 
P  for  any  given  project  size,  and  the  P  that 
satisfies  the  relationship  is  termed  P* .  The 
P*  associated  with  the  residuals  is  also  the 
P*  for  the  total  markup  for  any  given  project 
size  since  A  +  CXK  is  constant.  The  optimum 
markup  to  bid  for  a  given  project  size  is 
therefore : 


M*  *  A  +  CXk  +  R(P*) 


(B.7) 


One  may  obtain  a  general  bidding  policy  over 
+  a  range  ‘of  project  sizes  by  calculating  M*  at, 

say,  20  levels  of  project  size,  and  then  fit¬ 
ting  these  M*  observations  as  a  function  of 
project  size  by  the  equation: 

I  M*  =  A*  +  C*Xk*  (B .  8) 


This  equation  represents  the  M-Star  Methodo¬ 
logy. 
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“0.04924500 
-0.05487700 
“0.04387500 
“0.07U33300 
“0.07631200 
“0.08186400 
“0.08703600 
-0.09184500 
-0.09638700 
-0.10043000 
-0.10462000 
-0.10838000 
-0.11193000 
“0.11529000 
-0.11847000 
-0.12149000 
-0.12435000 
“0.12710000 
-0.12970000 
-0.13220000 


Table  C . 1  Lambda  Parameters 
(continued) 


LANBDA4 
1.54010000 
2.31240000 
2.75070000 
0.41942000 
0.29943000 
0.21565000 
0 . 15854000 
0.11675000 
0.08428800 
0.05805900 
0.03*27300 
0.01780200 
0.00956400 
-0.00030900 
-0.00527800 
-0.01199300 
-0.02423500 
“0.03512800 
“0.04489400 
-0.05371000 
-0.04 1 71300 
-0.04901700 
-0.07571300 
“0.08187900 
-0.08757800 
•0.09286300 
-0.09778000 
-0.10237000 
-0.10466000 
-0.11068000 
-0.11447000 
-0.11803000 
-0.12139000 
-0.12457000 
-0.12759000 
-0.13045000 
-0.13320000 
-0.13570000 
-0.13820000 


AIPHA3 


ALPHA4 

LAMB0A1 

LAhBDA2 

1.8 

0.153 

0.58050000 

2.0 

0.159 

0.53860000 

2.2 

0.174 

0.50420000 

2.4 

-0.706 

0 . 29 1 90000 

2.6 

-0.471 

0.27180000 

2.8 

-0.322 

0.23740000 

3.0 

-0.237 

0.19830000 

3.2 

-0.187 

0.15990000 

3.4 

-0.154 

0.12400000 

3 .6 

-0.132 

0.09080000 

3.8 

-0.115 

0.06010000 

4.0 

-0.103 

0.03190000 

4.1 

-0.097 

0.01850000 

4.2 

-0.093 

0.00570000 

4.3 

-0.089 

-0.00664100 

4.4 

-0.085 

-0.01850000 

4 .6 

-0.079 

-0.04110000 

4.8 

-0.073 

-0.06210000 

5.0 

-0.069 

-0.08180000 

5.2 

-0.065 

-0.10030000 

5.4 

-0.061 

-0.11760000 

5 .6 

-0.058 

-0.13390000 

5.8 

-0.055 

-0.14940000 

6.0 

-0.053 

-0.16400000 

6.2 

-0.051 

-0. 17780000 

6.4 

-0.049 

-0.19090000 

6.6 

-0.047 

-0.20340000 

6.8 

-0.046 

-0.21530000 

7.0 

-0.044 

-0.22660000 

7.2 

-0.043 

-0.23740000 

7.4 

-0.041 

-0.24780000 

7.6 

-0.040 

-0.25770000 

7.8 

-0.039 

-0.26720000 

8.0 

-0.038 

-0.27630000 

8.2 

-0.037 

-0.28500000 

8.4 

-0.036 

-0.29340000 

8.6 

-0.035 

-0.30140000 

8.8 

-0.035 

-0.30920000 

9.0 

-0.034 

-0.31680000 

9.2 

-0.034 

-0.32410000 

0.20 


LAMB0A3 

LAMBPA4 

2.24900000 

1.52070000 

3.28280000 

2.13200000 

4.14O3OOO0 

2.54490000 

0.10126000 

0.42460000 

0.12334000 

0.31197000 

0.12211000 

0.22731000 

0. 1 0652000 

0.16720000 

0.08658800 

0.12301000 

0.06667800 

0.08888700 

0.04819000 

0.04150100 

0.03144700 

0.03888600 

0.01640200 

0.01980300 

0.00946700 

0.01130000 

0.00289400 

0.00342900 

-0.00334200 

-0.00392900 

-0.00926100 

-0.01031200 

-0.02023600 

-0 . 02333  60  U 

-0.03018500 

-0.03445500 

-0.03924100 

-0.04440400 

-0.04751700 

-0.05337000 

-0.05510900 

-0.06149700 

-0.06209800 

-0.06890600 

-0.06855500 

-0.07569000 

-0.07453900 

-0.08193100 

-0.08010000 

-0.08769400 

-0.08528300 

-0.09303300 

-0.09012700 

-0.09799800 

-0.09466400 

-0.10263000 

-0.09892300 

-0.10695000 

-0.10293000 

-0.11101000 

-0.10671000 

-0.11 402000 

-0.11028000 

-0.11841000 

-0.1 1365000 

-0.12179000 

-0.1 1685000 

-0.12499000 

-0.1 1988000 

-0.12802000 

-0.12277000 

-0.13090000 

-0.12550000 

-0.13360000 

-0.12810000 

-0.13620000 

-0.13060000 

-0. 1 3870000 

-0.13300000 

-0.14110000 

ALPHA3 


ALPHA4 

LAHB0A1 

LAhB0A2 

2.0 

0.197 

0.54300000 

2.2 

0.214 

0.50740000 

2.4 

-0.789 

0.28200000 

2.6 

-0.558 

0.26490000 

2.8 

-0.398 

0.23490000 

3.0 

-0.299 

0.19870000 

3.2 

-0.236 

0.16190000 

3.4 

-0.196 

0.12660000 

3.6 

-0.167 

0.09370000 

3.8 

-0.146 

0.06330000 

4.0 

-0.130 

0.03500000 

4.1 

-0.126 

0.02170000 

4.2 

-0.118 

0.00890000 

4.3 

-0.113 

-0.00347600 

4.4 

-0.108 

-0.01540000 

4.6 

-0.100 

-0.03800000 

4.8 

-0.093 

-0.05910000 

5.0 

-0.087 

-0.07890000 

5.2 

-0.082 

-0.09740000 

5.4 

-0.077 

-0.11480000 

5.6 

-0.073 

-0.13120000 

5.8 

-0.070 

-0.14670000 

6.0 

-0.067 

-0.16140000 

6.2 

-0.064 

-0.17530000 

6.4 

-0.062 

-0.18840000 

6.6 

-0.059 

-0.20100000 

6.8 

-0.057 

-0.21290000 

7.0 

-0.055 

-0.22430000 

7.2 

-0.054 

-0.23520000 

7.4 

-0.052 

-0.24550000 

7.6 

-0.051 

-0.25550000 

7.8 

-0.049 

-0.26500000 

8.0 

-0.048 

-0.27420000 

8.2 

-0.047 

-0.28290000 

8.4 

-0.046 

-0.29140000 

8.6 

-0.045 

-0.29950000 

8.8 

-0.044 

-0.30720000 

9.0 

-0.043 

-0.31470000 

9.2 

-0.042 

-0.32200000 

*  0.25 


LAMB0A3 

LArtBDA4 

3.31000000 

1 .94780000 

4.25540000 

2. 34650000 

0.08435700 

0.42936000 

0.10617000 

0.32259000 

0.10988000 

0.23852000 

0.09955600 

0.17631000 

0.08312400 

0.  1  2997000 

0.06531900 

0.09420100 

0.04810100 

O.O456O6O0 

0.03214500 

0.04210500 

0.01759700 

0.02235400 

0.01080000 

0.01360000 

0.00440800 

0.00546700 

-0.00171300 

-0.00210300 

-0.00754000 

-0.00917500 

-0.01838200 

-0.02201900 

-0.02824800 

•0.03339500 

-0.03725300 

-0.04355500 

-0.04550100 

-0.05269500 

-0.05308100 

-0.06096900 

-0.06007000 

-0.06850000 

-0.06653400 

-0.07539000 

-0.07253100 

-0.08172000 

-0.07810900 

-0.08756000 

-0.08331200 

-0.09296*00 

-0. 0881 7600 

-0.09798900 

-0.09273500 

-0.10267000 

-0.09701700 

-0  .  10704000 

-0.10105000 

-0.11114000 

-0.10485000 

-0.11498000 

-0.10844000 

-0. 11860000 

-0.11184000 

-0.12201000 

-0.11506000 

-0.12524000 

-0.11811000 

-0.12829000 

-0.12102000 

-0.13118000 

-0.12379000 

-0.13393000 

-0.12640000 

-0. 13650C0O 

-0.12890000 

-0.13900000 

-0.13130000 

-0.14140000 

Table  C.l 


Lambda  Parameters 
( continued ) 


ALPHA4 

t Art  BOA  1  IAM8DA2 

2.0 

0.235  0.54860000 

2.2 

0.252  0.51150000 

2.4 

-0.071  0.27320000 

2.6 

-0.642  0.25860000 

2.8 

-0.474  0.23230000 

3.0 

-0.362  0.19920000 

3.2 

-0.288  0.16410000 

3.4 

-0.239  0.12980000 

3.6 

-0.204  0.09730000 

3.8 

‘0.179  0.06710000 

4.0 

-0.159  0.03890000 

4.2 

-0.144  0.01280000 

4.3 

-0.130  0.00078900 

4.4 

-0.131  -0.01 160000 

4.5 

-0.129  -0.02310000 

4.6 

-0.121  -0.03430000 

4.8 

-0.113  -0.05540000 

5.0 

-0.105  -0.07530000 

5.2 

-0.099  -0.09390000 

5.4 

-0.094  -0.11140000 

5.6 

-0.089  -0.12790000 

5.8 

-0.085  -0.14350000 

6.0 

-0.081  -0.15820000 

6.2 

-0.078  -0.17220000 

6.4 

-0.075  -0.18540000 

6.6 

-0.072  -0.19000000 

6.8 

-0.069  -0.21000000 

7.0 

-0.067  -0.22150000 

7.2 

-0.065  -0.23240000 

7.4 

-0.063  -0.24280000 

7.6 

-0.061  -0.25280000 

7.8 

-0.060  -0.26240000 

8.0 

-0.058  -0.27160000 

8.2 

-0.057  -0.28040000 

8.4 

-0.055  -0.28090000 

8.6 

-0.054  -0.29710000 

8.8 

-0.053  -0.30500000 

9.0 

-0.052  -0.31250000 

9.2 

-0.051  -0.31970000 

0.30  - 

L Art  BO  A3 
3.28050000 
4.32460000 
0.06949200 
0.09108500 
0.09830400 
0.09254200 
0.07956400 
0.06397500 
0.04813000 
0.03303000 
0.01902100 
0.00617500 
0.00038000 
-0.00555400 
-0.01100000 
-0.01625900 
-0.02604100 
-0.03500000 
-0.04322500 
-0.05080000 
-0.05779500 
-0.06427400 
-0.07029000 
-0.07589200 
-0.08112100 
■0.08601300 
■0.09060100 
•0.09491200 
0.09897100 
0.10280000 
0.10642000 
0.10984000 
0.1 1309000 
0.11617000 
0.11911000 
0.12190000 
0. 12460000 
0.12710000 
0.12950000 


Table 


LAHBDA4  ALPHA4 

1.75790000  2.0 

2.15530000  2.2 

0.43477000  2.4 

0.33243000  2.6 

0.24952000  2.8 

0.18593000  3.0 

0. 1 3768000  3.2 

0. 1 0U280O0  3.4 

0.02043200  3.6 

0.04597400  3.8 

0.02549100  4.0 

0.00803500  4.2 

0.00048900  4.3 

-0.00705700  4.4 

-0.01390000  4.5 

-0.02026200  <.4 

-0.03193100  4.8 

-0.04233300  5.0 

-0.05167400  5.2 

-0.06011700  5.4 

-0.06779100  5.6 

-0.07480300  5.8 

-0.08123900  6.0 

-0.087171 OO  6.2 

-0.09265200  6.4 

-0.09774900  6.6 

-0.10249000  6.8 

-0.10692000  7.0 

0.11106000  7.2 

0.1  1  495000  7  a 4 

0.1  1861000  7^ 

0.12205000  7.0 

0.12531000  b*.  0 

0.12839000  0.2 

0.13131000  8.4 

0.  1 3408000  8.6 

0.13670000  8.8 

0.13920000  9.0 

0.14160000  9.2 


AIPHA3 


1  LArtfiOAl 

L  Art  BB A2 

0.273 

0.55570000 

0.289 

0.51650000 

-0.955 

0.26530000 

-0.725 

0.25270000 

-0.550 

0.22980000 

-0.427 

0.19960000 

-0.343 

0.16650000 

-0.285 

0.13330000 

-0.243 

0.10140000 

-0.213 

0.07140000 

-0.189 

0.04340000 

-0.171 

0.01730000 

-0.163 

0.00487000 

-0.156 

-0.00710000 

-0.151 

-0.01870000 

-0.  144 

-0.02990000 

-0.133 

-0.05110000 

-0.125 

-0.07110000 

-0.117 

-0.08980000 

-0.111 

-0.10740000 

-0.105 

-0.12400000 

-0.100 

-0.13960000 

-0.096 

-0.15450000 

-0.092 

-0.16850000 

-0.088 

-0.16180000 

-0.085 

-0.19450000 

-0.082 

-0.20660000 

-0.079 

-0.21810000 

-0.077 

-0.22910000 

-0.074 

-0.23960000 

-0.072 

-0.24970000 

-0.070 

-0.25930000 

-0.068 

-0.26860000 

-0.067 

-0 . 277400C0 

-0.065 

-0.28600000 

-0.063 

-0.29420000 

-0.062 

-0.30200000 

-0.060 

-0.30960000 

-0.059 

-0.31720000 

=  U  .  J  j  - 

L  A  rt  B  2)  A  3 

IAHBDA4 

3.17470000 

1.55620000 

4.33920000 

1.94970000 

0.05590100 

0.44146000 

0.07745400 

0.34228000 

0.08727100 

0.24060000 

0.08542600 

0.19611000 

0.07580900 

0.14419000 

0.06254100 

0.10719000 

0.0481  8  f  OO 

0.07602100 

0.03403400 

0.05053600 

0.02062400 

0.02925100 

0.00815800 

0.01116400 

0.00229300 

0.00309000 

-0.00333200 

-0.00443100 

-0.00872300 

-0.01150000 

-0.01388900 

-0.01804200 

-0.02358300 

-0.03004400 

-0. 03249300 

-0.04072100 

-0.04069900 

-0.05029200 

-0.04827300 

‘0. O589290U 

-0.05528000 

-0.06676900 

-0.06177800 

-0.07392300 

-0.06782100 

-0.08048200 

-0.07345200 

-0.08652000 

-0.07871300 

-0.09209900 

-0.08363900 

-0.09727400 

-0.08826100 

-0.10209000 

-0.09260600 

-0.10658000 

-0.09669900 

-0.11078000 

-0.10056000 

-0.11472000 

-0.10421000 

-0.11842000 

-0.10767000 

-0.12191000 

-0.11095000 

-0.12520000 

-0.11406000 

-0.12832000 

-0.11702000 

-0.13127000 

-0.11984000 

-0.13407000 

-0.12250000 

-0.13670000 

-0.12510000 

-0.13920000 

-0.12760000 

-0.14170000 

c.  1 


Lambda  Parameters 
( continued) 


ro 


ALPHA3  *  0.40 


ALPHA4 

LAH8DA 1 

LAH80A2 

LAH80A3 

2.0 

0.313 

0.56430000 

2.94950000 

2.2 

0.325 

0.52260000 

4.29000000 

2.4 

-1.043 

0.25800000 

0.04305000 

2.6 

-0.808 

0.24730000 

0.06480500 

2.8 

-0.628 

0.22730000 

0.07665900 

3.0 

-0.495 

0.20000000 

0.07816600 

3.2 

-0.400 

0.16900000 

0.07177100 

3.4 

-0.333 

0.13710000 

0.06091200 

3.6 

-0.285 

0.10600000 

0.04818200 

3.8 

-0.249 

0.07640000 

0.03508100 

4.0 

-0.221 

0.04850000 

0.02234600 

4.2 

-0.199 

0.02240000 

0.01031100 

4.3 

-0.190 

0.01000000 

0.00459700 

4.4 

-0.182 

-0.00039700 

-0.00018200 

4.5 

-0.174 

-0.01360000 

-0.00420400 

4.6 

-0.167 

-0.02480000 

-0.01129600 

4.8 

-0.155 

-0.04620000 

-0.02089100 

5.0 

-0.145 

-0.06620000 

-0.02974900 

5.2 

-0.136 

-0.08500000 

-0.03793400 

5.4 

-0.129 

-0.10270000 

-0.04550700 

5.6 

-0.122 

-0.11940000 

-0.05252900 

5.8 

-0.116 

-0.13520000 

-0.05905200 

6.0 

-0.111 

-0.15010000 

-0.06512500 

6.2 

-0.106 

-0.16430000 

-0.07079100 

6.4 

-0.102 

-0.17770000 

-0.07609000 

6.6 

-0.098 

-0.19050000 

-0.08105500 

6.8 

-0.095 

-0.20270000 

-0.08571600 

7.0 

-0.091 

-0.21430000 

-0.09010100 

7.2 

-0.088 

-0.22530000 

-0.09423300 

7.4 

-0.086 

-0.23590000 

-0.09813400 

7.6 

-0.083 

-0.24610000 

-0.10182000 

7.8 

-0.081 

-0.25580000 

-0. 105  320  00 

8.0 

-0.079 

-0.26510000 

-0.10863000 

8.2 

-0.077 

-0.27400000 

-0. 1 t 178000 

8.4 

-0.075 

-0.28260000 

-0.11477000 

8.6 

-0.073 

-0.29090000 

-0.11763000 

8.8 

-0.072 

-0.29880000 

-0.12035000 

9.0 

-0.070 

-0.30640000 

-0.12290000 

9.2 

-0.069 

-0.31390000 

-0.12540000 

9.4 

-0.067 

-0.32100000 

-0.12780000 

Tabl 


LArtB0A4 

ALPHA4 

LAH80A1 

LAM6DA2 

£  0.45  - 

L  API  B  9  A  3 

LA  fill)  A  4 

1.32530000 

2.0 

0.381 

0.57320000 

2.23920000 

0.89646000 

1.78740000 

2.2 

0.359 

0.52980000 

4.16540000 

1 .60400000 

0.44996000 

2.4 

-1.138 

0.25110000 

0.03050600 

0.46082000 

0.35268000 

2.6 

-0.895 

0.24230000 

0.05276800 

0.36411000 

0.27200000 

2.8 

-0.70/ 

0.22480000 

0.06633100 

0.28400000 

0.20691000 

3.0 

-0.565 

0.20040000 

0.07071900 

0.21841000 

0.15552000 

3.2 

-0.461 

0.17160000 

0.06737000 

0.16572000 

0.11495000 

3.4 

-0.385 

0.14120000 

0.05898100 

0.12362000 

0.08242400 

3.6 

-0.329 

0.11100000 

0.04800700 

0.01968900 

0.05583900 

3.8 

-0.287 

0.08180000 

0.03608300 

0.06193000 

0.0336/700 

4.0 

-0.255 

0.05420000 

0.02412000 

0.03881200 

0.01489100 

4.2 

-0.229 

0.02820000 

0.01258200 

0.01925400 

0.00652100 

4.3 

-0.221 

0.01580000 

0.00704500 

0.01060000 

-0.00025400 

4.4 

-0.208 

0.00410200 

0.00183300 

0.00269100 

-0.00853300 

4.5 

-0.200 

-0.00786100 

-0.00350500 

-0.00506500 

-0.01533400 

4.6 

-0.192 

-0.01910000 

-0.00851100 

-0.01210900 

-0.02771 100 

4.8 

-0.178 

-0.04060000 

-0.01798900 

-0.02490800 

-0.03870000 

5.0 

-0.166 

-0.06070000 

-0.02678400 

-0.03625000 

-0.04853300 

5.2 

-0.156 

-0.07970000 

-0.03494200 

-0.04637900 

-0.05739200 

5.4 

-0.147 

-0.09750000 

-0.04251400 

-0.05548900 

-0.06542100 

5.6 

-0.139 

-0.11430000 

-0.04955000 

-0.06373400 

'0.07273800 

5.8 

-0.133 

-0.13020000 

-0.05610000 

-0.07123700 

-0.07943800 

6.0 

-0.127 

-0.14530000 

-0.06220700 

-0.07809800 

-0.08559900 

6.2 

-0.121 

-0.15950000 

-0.06791200 

-0.08440000 

-0.09128700 

6.4 

-0.116 

-0.17310000 

-0.07325300 

-0.09021100 

-0.09655700 

6.6 

-0.112 

-0.18590000 

-0.07826100 

-0.09559000 

-0.10145000 

6.8 

-0.108 

-0.19820000 

-0.08296600 

-0.10059000 

-0.10602000 

7.0 

-0.104 

-0.20990000 

-0.08739500 

-0, 10524000 

-0.11029000 

7.2 

-0.101 

-0.22110000 

-0.09157100 

-0.10958000 

-0 . t 1429000 

7.4 

-0.098 

-0.23170000 

-0.09551600 

-0.11365000 

'0. 11805000 

7.6 

-0.095 

-0.24190000 

-0.09924700 

-0.11747000 

-0.12158000 

7.8 

-0.092 

-0.25170000 

-0.10278000 

-0.12107000 

-0.12492000 

8.0 

-0.090 

-0.26110000 

-0.10614000 

-0.12446000 

-0.12808000 

8.2 

-0.087 

-0.27010000 

-0.10932000 

-0.12766000 

-0.13106000 

6.4 

-0.085 

-0.27880000 

-0.11236000 

-0.13069000 

-0.13389000 

8.6 

-0.083 

-0.28710000 

-0.11524000 

-0.13356000 

-0.13658000 

8.8 

-0.081 

-0.29510000 

-0.11800000 

-0. 1 3628000 

-0.13910000 

9.0 

-0.080 

-0.30290000 

-0.12060000 

-0.13890000 

-0.14160000 

9.2 

-0.078 

-0.31020000 

-0.12310000 

-0.14130000 

-0.14390000 

9.4 

-0.076 

-0.31760000 

-0.12560000 

-0.14370000 

C.l  Lambda  Parameters 
( cont inued ) 


f 


ALPHAS  *  0.50 


ALPHA4 

LAMB0A1 

LAMB0A2 

LAMB0A3 

2.2 

0.395 

0.53840000 

3.94480000 

2.4 

-1.245 

0.24450000 

0.01782800 

2.6 

-0.986 

0.23760000 

0.04102100 

2.8 

-0.790 

0.22250000 

0.054141 00 

3.0 

-0.639 

0.20070000 

0.06304000 

3.2 

-0.525 

0.17420000 

0.06254100 

3.4 

-0.440 

0.14550000 

0.05665000 

3.6 

-0.376 

0.11630000 

0.04755900 

3.8 

-0.328 

0.08770000 

0.03694900 

4.0 

-0.291 

0 . 06040000 

0.02586700 

4.2 

-0.242 

0.03450000 

0.01491100 

4.3 

-0.248 

0.02210000 

0.00958200 

4.4 

-0.238 

0.01010000 

0.00438300 

4.5 

-0.228 

-0.00161200 

-0.00070000 

4.6 

-0.219 

-0.01280000 

-0.00557000 

4.8 

-0.202 

-0.03440000 

-0.01490400 

5.0 

-0.189 

-0.05460000 

-0.02361900 

5.2 

-0.177 

“0.07370000 

-0.03174000 

5.4 

-0.167 

-0.09170000 

-0.03930500 

5.6 

-0.158 

-0.10860000 

-0.04635400 

5.8 

-0.150 

-0.12470000 

-0.05293000 

6.0 

-0.143 

-0.13980000 

-0.05907200 

6.2 

-0.137 

-0.15420000 

-0.06481900 

4.4 

-0.131 

-0.16790000 

-0.07020400 

6.6 

-0.126 

-0.18090000 

-0.07526000 

6.8 

-0.122 

-0.19320000 

-0.08001400 

7.0 

-0.117 

-0.20500000 

-0.08449100 

7.2 

-0.114 

-0.21630000 

-0.08871500 

7.4 

-0.110 

-0.22710000 

-0.09270700 

7.6 

-0.107 

-0.23740000 

-0.09648400 

7.8 

-0.104 

-0.24720000 

-0.10006000 

8.0 

-0.101 

-0.25670000 

-0.10346000 

8.2 

-0.098 

-0.26580000 

-0.10669000 

8.4 

-0.096 

-0.27450000 

-0.10976000 

8.6 

-0.094 

-0.28290000 

-0.11269000 

8.8 

-0.091 

-0.29100000 

-0.11549000 

9.0 

-0.089 

-0.29860000 

-0.11810000 

9.2 

-0.088 

-0.30640000 

-0.12070000 

9.4 

-0.086 

-0.31340000 

-0.12310000 

9.6 

-0.084 

-0.32060000 

-0.12550000 

i 


LAMBDA4 
1.41230000 
0.47479000 
0.37702000 
0.2*686000 
0.23071000 
0.17681000 
0.13323000 
0.09786300 
0.06885700 
0.04470300 
0.02429500 
0.01520000 
0.00681500 
“0.00106600 
-0.00033400 
-0.02160800 
-0.03334500 
-0.04380900 
“0.05320400 
“0.04169200 
-0.04940500 
-0.07644900 
-0.08291100 
-0.08886400 
-0.0*436700 
-0.09947300 
-0. 10422000 
“0.10866000 
-0.11281000 
-0.11670000 
-0.12036000 
-0.12381000 
-0.12706000 
-0.13014000 
-0.13306000 
-0.13582000 
-0.13840000 
-0.14100000 
-0. 14330000 
-0.14560000 


ALPHA4  LAMBDA! 

2.2  0.433 

2.4  -1.370 

2.6  -1.086 

2.8  -0.877 

3.0  -0.716 

3.2  -0.593 

3.4  -0.499 

3.6  -0.427 

3.8  -0.373 

4.0  -0.330 

4.2  -0.296 

4 • 4  -0.269 

4.5  -0.257 

4.6  -0.247 

4.7  -0.237 

4 . 8  -0.228 

5.0  -0.212 

5.2  -0.199 

5.4  -0.187 

5.6  -0.177 

5.8  -0.166 

6.0  -0.160 

6.2  -0.153 

6.4  -0.147 

6.6  -0.141 

6.8  -0.136 

7.0  -U.131 

2.2  -0.127 

2.4  -0.123 

2.6  -0.119 

2.8  -0.116 

8.0  -0.113 

0.2  -0.110 

0.4  -0.107 

0.6  -0.104 

8.0  -0.102 

9.0  -0.100 

9.2  -0.097 

9.4  -0.095 

9.6  -0.094 


ALPHA3 
LAMBDA? 
0.54840000 
0.23790000 
0.23310000 
0.22030000 
0.20090000 
0. 17670000 
0.14970000 
0.12170000 
0.09390000 
0.06700000 
0.04130000 
0.01700000 
0.00535500 
-0.00600000 
-0.01690000 
-0.02760000 
“0.04790000 
-0.06710000 
-0.08520000 
-0.10230000 
“0.11850000 
-0.13380000 
“0.14830000 
-0.16210000 
“0. 17530000 
-0. 18770000 
-0.19970000 
-0.21100000 
-0.22190000 
-0.23230000 
-0.24230000 
-0.25180000 
-0.26100000 
-0.26980000 
-0.27830000 
-0.28650000 
-0.29430000 
“0.30190000 
-0.30920000 
-0.31640000 


8  0.55  - 

LAMBDAS 

3.59000000 

0.00446300 

0.02924300 

0.04592900 

0.05507400 

0.05722600 

0.05382400 

0.04672800 

0.03757400 

0.02750200 

0.01722600 

0.00714900 

0.00225800 

-0.00251500 

-0.00716000 

-0.01167000 

-0.02028000 

-0.02834800 

-0.03589500 

-0.04295100 

-0.04955000 

-0.05572700 

-0.06151600 

“0.06694900 

-0.07205400 

-0.07685900 

-0.08138800 

-0.08566400 

-0.08970700 

-0.09353400 

-0.09716300 

-0.10061000 

-0.10388000 

-0.10700000 

-0.10997000 

“0.11281000 

-0.11550000 

-0.11810000 

-0.12060000 

-0.12300000 


e  C.]  Lambda  Parameters 
( continued ) 


LAMBDA4 
1.19530000 
0.49307000 
0.39200300 
0.31090000 
0.24394000 
0 . 1 8087000 
0.14382000 
0.10499000 
0.07467100 
0.05139700 
0.03005900 
0.01181200 
0.00364400 
-0 .00397500 
-0.01110000 
-0.01777900 
“0.02996000 
-0.04080000 
-0.05051400 
-0.05927800 
•0.06722800 
“0.07447900 
-0.08112200 
-0.08723300 
-0.09287100 
-0.09810900 
-0.10297000 
-0.10751000 
-0.11175000 
-0.11572000 
“0.11945000 
•0. 12297000 
“0.12629000 
“0.12942000 
“0.13239000 
'0.13520000 
0.13790000 
0.14040000 
0.14280000 
0.14520000 


ALPHA*  LAH00A1 

2.2  0. *99 

2.*  0. *82 

2.6  -1.190 


2.8  -0.972 

3.0  -0.799 

3.2  -0.665 

3. *  -0.562 

3.6  -0. *82 

3.0  -0. *20 

*.0  -0.372 

4.2  -0.33* 

4. *  -0.303 

4.5  -0.289 

4.6  -0.277 

4.7  -0.266 

4.8  -0.256 

5-0  -0.238 

5.2  -0.223 

5.4  -0.209 

5.6  -0.198 

5.8  -0.188 

6.0  -0.179 

6.2  -0.171 

6.4  -0.163 

6.6  -0.157 

6.8  -0.151 

2.0  -0.1*6 

2.2  -0.1*1 

2.4  -0.136 

2.6  -0.132 

2.8  -0.128 

8.0  -0.125 

0.2  -0.121 

0.4  -0.118 

0.6  -0.115 

0.0  -0.113 

9.0  -0.110 

9.2  -0.108 

9.4  -0.105 

9.6  -0.103 


ALPHA3 
LAMB0A2 
0.55820000 
0.52030000 
0.22860000 
0.21800000 
0.20100000 
0.17910000 
0.15*00000 
0.127*0000 
0.100*0000 
0.07*00000 
0.0*860000 
0.02**0000 
0.01280000 
0.00150000 
“0.00953100 
“0.02020000 
-0.0*070000 
-0.06000000 
-0.07820000 
-0.09550000 
“0.1 1 180000 
“0.12730000 
‘0.1*190000 
“0.15590000 
“0.16910000 
*0.18170000 
“0.19380000 
“0.20530000 
*0.21630000 
‘0.22680000 
-0.23680000 
‘0.2*650000 
-0.25580000 
-0.26*70000 
“0.27320000 
“0.28150000 
-0.289*0000 
“0.29700000 
“0.30*50000 
-0.31160000 


5  0.60  - 

LAMB0A3 
2.82520000 
*.77890000 
0.01705500 
0.03551000 
0.0*675000 
0.05137200 
0.050*2100 
0.0*5*1000 
0.03785100 
0.02092800 
0.01 9**900 
0.00991100 
0.00521500 
0.00061100 
-0.00391600 
-0.00832600 
“0.01679800 
“0.02*79000 
-0.03230300 
“0.03935500 
“0.0*597200 
“0.05218100 
-0.05801 100 
“0.063*9000 
-0.0686*700 
“0.07350500 
“0.07808900 
“0.082*2000 
-0.08651700 
-0.09039800 
“0.09*08000 
“0.09757600 
“0.10090000 
-0.10*07000 
-0.10709000 
-0.10997000 
-0. 11272000 
“0.11530000 
-0.11790000 
“0.12030000 


LAABBA* 
0.852**000 
1.36010000 
0.40983000 
0.32653000 
0.25035000 
0.20197000 
0.15545000 
0.11713000 
0.005*2200 
0.0509*800 
0.03659600 
0.01750200 
0.00896500 
0.00102400 
-0.006*2500 
-0.01338600 
-0.02406500 
-0.03732500 
-0.0*739800 
-0.056*7000 
-0.06*68800 
-0.07217000 
-0.07901700 
-0.08530700 
-0.091 1 1000 
“0.096*0300 
-0.101*7000 
-0. 1 0612000 
-0. 110*6000 
“0. 1 1453000 
“0.1 1835000 
-0.1219*000 
-0.12532000 
"0. 12852000 
-0.13155000 
“0.13**1000 
“0.1371*000 
-0.1 3970000 
-0.14220000 
-0. 14*50000 


Table  C. l 


ALPHA* 

L  Art 0OA 1  LAMB0A2 

2.* 

0.516  0.530*0000 

2.6 

*1.329  0.22*00000 

2.8 

-1.076  0.21570000 

3.0 

‘0.889  0.20100000 

3.2 

“0.7**  0.10130000 

3.  * 

“0.630  0.15810000 

3.6 

-0.5*2  0.13300000 

3.8 

-0.472  0.10710000 

*.0 

-0.418  0.081*0000 

*.2 

“0.37*  0.05630000 

*.* 

“0.339  0.03230000 

*.5 

-0.32*  -0.02070000 

*.6 

“0.298  -0.000*0000 

*.7 

-0.297  -0.00159300 

*.0 

“0.285  -0.01230000 

5.0 

“0.265  -0.03290000 

5.2 

“0.2*8  -0.05230000 

5.* 

“0.233  -0.07070000 

5.6 

-0.220  -0.00810000 

5.8 

“0.200  -0.10*60000 

6.0 

“0.198  -0.12020000 

6.2 

‘0.189  -0.13500000 

6.* 

-0.181  -0.1*910000 

6.6 

-0.17*  -0.16250000 

6.8 

‘0.167  -0.17520000 

7.0 

“0.161  -0.187*0000 

7.2 

‘0.155  -0.19900000 

7.* 

-0.150  -0.21010000 

7.6 

“0.1*6  -0.22080000 

7.8 

-0.1*1  -0.23090000 

0.0 

“0.137  -0.2*070000 

0.2 

-0.13*  -0.25010000 

8.* 

-0.130  -0.25910000 

8.6 

-0. 127  -0.26780000 

8.8 

“0.12*  -0.27610000 

9.0 

-0.121  -0.20*10000 

9.2 

-0.119  -0.29190000 

9.* 

-0.116  -0.299*0000 

9.6 

-0.11*  -0.30650000 

9.8 

-0.112  -0.31360000 

*  0.65  ----- 
LAM0OA3 
*.36130000 
0.00390800 
0.02*6*100 
0.03796700 
0.0**92*00 
0.0*636600 
0.0*3*9900 
0.03766700 
0.0300*000 
0.021*0700 
0.01250900 
0.00813700 
-0.00016100 
“0.00063*00 
“0.00*92100 
“0.01321  100 
-0.02109500 
-0.02855300 
-0.03558700 
“0.0*221000 
“O.O*0***OO 
-0.05*31100 
-0.05903600 
-0.0650**00 
“0.06995700 
“0.07*59700 
-0.07898*00 
“0.08313000 
-0.08707600 
-0.09001200 
‘0.09*36300 
-0.0977*100 
“0.10096000 
-0.10*03000 
-0.10695000 
-0.10975000 
-0.1 12*3000 
-0.11500000 
-0.117*0000 
-0.1 1980000 


Lambda* 

1 .17300000 
0. *3  1  02000 
0.3**31000 
0.27*16000 
0.2162*000 
0.16819000 
0. 12832000 
0.09516600 
0.047*0900 
0.04395700 
0.02392700 
0.01500000 
-0.00020200 
“0.001 10600 
-0.00839100 
“0.02142*00 
-0.03335*00 
-0.0*302900 
-0.0532*600 
“0.06176300 
“0.04950600 
-0.07658100 
“0.08307200 
“0.00905300 
“0.09*50*00 
-0.09971 500 
-0.10**9000 
*0.10895000 
“0.11312000 
“0.1 1703000 
-0.1 2070000 
-0.12*17000 
-0.127**000 
*0.13053000 
'0. 133*6000 
-0.13623000 
-0.13887000 
“0.1*140000 
-0.1*380000 
-0.1*610000 


Lambda  Parameters 
( continued ) 


I 


+ 


NJ 

iO 


ALPHA4 

LAH6DA1 

LAH80A2 

L AHB8A3 

LAHBPA4 

ALPHA4 

2.4 

0.563 

0.54110000 

3.71660000 

0.93547000 

2.6 

2.6 

0.569 

0.50720000 

5.48890000 

1  .25770000 

2.8 

2.8 

-1.194 

0.21320000 

0.01296700 

0.36513000 

3.0 

3.0 

-0.987 

0.20080000 

0.02857700 

0.29179000 

3.2 

3.2 

-0.828 

0.18330000 

0.03781 100 

0.23188000 

3.4 

3.4 

-0.704 

0.16210000 

0.04158800 

0.18212000 

3.6 

3.6 

-0.606 

0.13860000 

0.04090300 

0.14063000 

3.8 

3.8 

-0.529 

0.11390000 

0.03691300 

0.10596000 

4.0 

4.0 

-0.467 

0.08890000 

0.03072400 

0.07684000 

4.2 

4.2 

-0.418 

0.06430000 

0.02324000 

0.05219900 

4.4 

4.6 

-0.345 

0.01780000 

0.00676700 

0.01302700 

4.6 

4.7 

-0.331 

0.00679900 

0.00260700 

0.00487200 

4.7 

4.8 

-0.318 

-0.00390000 

-0.00151 200 

-0.00275000 

4.8 

4.9 

-0.305 

-0.01440000 

-0.00557400 

-0.00989300 

4.9 

5.0 

-0.294 

-0.02450000 

-0.00956500 

-0.0165990U 

5.0 

5.2 

-0.275 

-0.04410000 

-0.01730000 

-0.02885400 

5.2 

5.4 

-0.258 

-0.06260000 

-0.02467300 

-0.03977700 

5.4 

5.6 

-0.243 

-0.08010000 

-0.03166700 

-0.04958100 

5.6 

5.8 

-0.230 

-0.09680000 

-0.03828200 

“0.05843100 

5.8 

6.0 

“0.219 

-0.11250000 

-0.04453000 

-0.06646600 

6.0 

6.2 

-0.208 

-0.12750000 

-0.05042800 

-0.07379600 

6.2 

6.4 

-0.199 

-0.14170000 

-0.05599500 

-0.08051200 

6.4 

6.6 

-0.191 

-0.15530000 

-0.06125100 

-0.08669200 

6.6 

6.8 

-0.184 

-0.16820000 

-0.06621700 

-0.09240000 

6.8 

7.0 

-0.177 

-0.18050000 

-0.07091400 

-0.09768900 

7.0 

7.2 

-0.171 

-0.19230000 

-0.07536000 

-0.10261000 

7.2 

7.4 

7.6 

7.8 
8.0 
8.2 

8.4 

8.6 

8.8 
9.0 
9.2 

9.4 
9.6 
9.8 


-0.165 

-0.160 

-0.155 

-0.151 

-0.147 

-0.143 

-0.139 

-0.136 

-0.133 

-0.130 

-0.127 

-0.125 

-0.122 


-0.20350000 

-0.21430000 

-0.22460000 

-0.23450000 

-0.24400000 

-0.25310000 

-0.26180000 

-0.27030000 

-0.27840000 

-0.28620000 

-0.29380000 

-0.30110000 

-0.30810000 


-0.07957200 
-0.08356800 
-0.08736300 
-0.09097100 
-0.09440400 
-0.09767600 
-0.10080000 
-0.10378000 
-0.10662000 
-0.10935000 
-0.11195000 
-0.11440000 
-0.1 1680000 


-0.10719000 
-0.11148000 
-0.11549000 
-0.11926000 
-0.12281000 
-0.12616000 
-0.12933000 
-0.13232000 
-0.13516000 
-0  .  1 3766000 
-0  .  14043000 
-0.14290000 
-0.14520000 


7.4 

7.6 

7.8 
8.0 
8.2 

8.4 

8.6 

8.8 
9.0 
9.2 

9.4 
9.6 
9.8 

10.0 


LAM8DA1 

0.601 

0.631 

-1.097 

-0.921 

-0.785 

-0.677 

-0.590 

-0.522 

-0.466 

-0.421 

-0.383 

-0.367 

-0.352 

-0.339 

-0.326 

-0.304 

-0.285 

-0.268 

-0.253 

-0.241 

-0.229 

-0.219 

-0.210 

-0.201 

-0.194 

-0.187 

-0.181 

-0.175 

-0.170 

-0.165 

-0.160 

-0.156 

-0.152 

-0.148 

-0.145 

-0.141 

-0.138 

-0.135 

-0.133 

-0.130 


-  ALPHA] 

LAHB0A2 
0.51 74O0U0 
0.46600000 
0.20030000 
0.18500000 
0.16580000 
0.14400000 
0.12060000 
0.09660000 
0.07260000 
0.04920000 
0.02660000 
0.01560000 
0.00490000 
-0.00550900 
-0.01570000 
-0.03530000 
-0.05400000 
-0.07160000 
-0.08840000 
-0.10440000 
-0.11950000 
-0.13390000 
-0.14760000 
-0.16070000 
-0.17320000 
-0.18510000 
-0. 1 9650000 
-0.20730000 
-0.21780000 
-0.22780000 
-0.23740000 
-0.24660000 
-0.25550000 
-0 . 26400000 
-0.27230000 
-0.28020000 
-0.28780000 
-0.29520000 
-0.30240000 
-0.30930000 


*  0.75  - . 

LAM60A3 

4.95390000 

6.60410000 

0.01834200 

0.02992700 

0.03601400 

0.03753100 

0.03547700 

0.03086700 

0.02459600 

0.01736200 

0.00966300 

0.00574900 

0.00183300 

-0.00206100 

-0.00591500 

-0.01345000 

-0.02070000 

-0.02762500 

-0.03421100 

-0.04045900 

-0.04637600 

-0.05197700 

-0.05727700 

-0.06229500 

-0.06704700 

-0.07155000 

-0.07582200 

-0.07987800 

-0.08373200 

-0.08739900 

-0.09089000 

-0.09421900 

-0.09739400 

-0.10043000 

-0.10333000 

-0.10610000 

-0.10876000 

-0.11131000 

-0.11375000 

-0.11610000 


LAHBDA4 
1 .01800000 
1 .27670000 
0.31189000 
0.24920000 
0.19743000 
0.15417000 
0.11788000 
0.08730500 
0.04138300 
0.03923900 
0.02016200 
0.01 16O0U0 
0.00358300 
-0.00391600 
-0.01094900 
-0.02376600 
-0.03521000 
-0.04544300 
-0.05466700 
“0.06302700 
-0.07064100 
-0.07760800 
-0.08401100 
-0.08991600 
-0.09538100 
-0.10046000 
-0. 1051 80U0 
-0.10960000 
-0.11373000 
-0.11761000 
-0.12126000 
-0.12470000 
-0.12794000 
-0.13101000 
-0.13392000 
-0.13668000 
-0.13931000 
-0.14181000 
-0.14419000 
-0.14650000 


Table  C. 1 


Lambda  Parameters 
( continued ) 


AIPHA3 


ALPHA4 

LAttBOAl  LAAB0A2 

2.6 

0.647  0.52790000 

2.8 

0.655  0.49760000 

3.0 

-1.225  0.19950000 

3.2 

“1.025  0.18650000 

3.4 

-0.873  0.16930000 

3.6 

“0.754  0.14920000 

3.8 

-0.658  0.12720000 

4.0 

“0.581  0.10420000 

4.2 

-0.518  0.08100000 

4.4 

-0.467  0.05810000 

4.6 

“0.425  0.03570000 

4.0 

-0.390  0.01420000 

4.9 

“0.375  0.00377000 

5.0 

“0.361  -0.00630000 

5.J 

-0.349  -0.01640000 

5.2 

-0.335  -0.02610000 

5.4 

-0.314  -0.04480000 

5.6 

“0.295  -0.06260000 

5.8 

-0.279  -0.07960000 

6.0 

“0.264  “0.09570000 

6.2 

-0.251  -0. 1 1100000 

6.4 

“0.240  -0.12560000 

6.6 

“0.230  -0.13940000 

6.0 

“0.220  -0.15270000 

7.0 

-0.212  -0.16530000 

7.2 

“0.204  -0.17740000 

7.4 

-0.197  -0.18890000 

7.6 

“0.191  -0.19990000 

7.8 

“0.185  -0.21050000 

8.0 

-0.179  -0.22070000 

8.2 

“0.174  -0.23040000 

8.4 

-0.170  -0.23970000 

8.6 

“0. 165  -0.24870000 

8.8 

’0.161  -0.25740000 

9.0 

-0.157  -0.26570000 

9.2 

-0.154  -0.27380000 

9.4 

-0.150  -0.28150000 

9.6 

-0.147  -0.28900000 

9.0 

-0.144  -0.29620000 

10.0 

-0.141  -0.30330000 

10.2 

“0.139  -0.31000000 

=  0.80 


LANB0A3 

LAhBDM 

4.20140000 

0.87379000 

6.04130000 

1 . 1 3630000 

0.00604900 

0.33555000 

0.021 10300 

0.26870000 

0.02954800 

0.21434000 

0.03329400 

0.16906000 

0.03325300 

0.13102000 

0.03034800 

0.09086000 

0.02543900 

0.07157800 

0.01924400 

0.04023400 

0.01231300 

0.02012400 

0.00503500 

0.01066000 

0.00135200 

0.00277000 

-0.00227800 

-0.00453100 

-0.00598100 

-0.01160000 

“0.00959800 

-0.01811000 

-0.01667500 

-0.03008000 

-0.02349500 

-0.04080100 

-0.03002500 

-0.05044000 

“0.03625200 

-0.05916200 

-0.04217400 

-0.06709300 

-0.04779800 

-0.07434000 

-0.05313500 

-0.00099000 

-0.05819800 

-0.00711500 

-0.06300200 

-0.09277700 

-0.06756200 

-0.09802900 

-0.07190000 

-0. 1 0290000 

-0.07600800 

-0. 10747000 

“0 . 07992300 

“0.  1  1  173000 

-0.08365000 

-0.  1  1573000 

-0.08720100 

-0.11949000 

-0.09058800 

-0.12302000 

“0.09382100 

-0. 12636000 

-0.09691000 

-0.12952000 

-0.09986400 

-0.13250000 

-0. 10269000 

-0.13534000 

“0.10540000 

“0 . 1  3803000 

-0.10800000 

“0.14059000 

-0.1 1049000 

-0.14302000 

-0.11290000 

“0.14540000 

-0.1 1520000 

0.14760000 

ALPHA4 

LAHB0A1  LA/1B0A2 

2.8 

0.686  0.50800000 

3.0 

0.715  0.48070000 

3.2 

“1.145  0.13750000 

3.4 

-0.972  0.17240000 

3.6 

-0.838  0.15410000 

3.8 

-0.732  0.13360000 

4.0 

-0.646  0.11180000 

4.2 

-0.576  0.08950000 

4.4 

-0.518  0.06710000 

4.6 

-0.471  0.04510000 

4.8 

-0.431  0.02380000 

4.9 

-0.413  0.03140000 

5.0 

-0.398  0.00350000 

5.1 

-0.383  -0.00670100 

5.2 

-0.370  -0.01640000 

5.4 

-0.345  -0.03520000 

5.6 

-0.324  -0.05320000 

5.8 

-0.306  -0.07020000 

6.0 

-0.289  -0.08650000 

6.2 

-0.275  -0. 10200000 

6.4 

-0.262  -0.1 1670000 

6.6 

-0.251  “0. 1 308U000 

6.8 

-0.240  -0.14420000 

7.0 

-0.231  -0.15700000 

7.2 

-0.223  -0.16920000 

7.4 

-0.215  -0.18090000 

7.6 

-0.208  -0.19210000 

7.8 

-0.201  -0.20280000 

8.0 

-0.195  -0.21310000 

8.2 

-0. 189  -0.22300000 

8.4 

-0.184  -0.23240000 

8.6 

-0. 179  -0.24160000 

8.6 

-0.175  -0.25030000 

9.0 

-0.171  -0.25880000 

9.2 

-0.167  -0.26690000 

9.4 

-0.163  -0.27400000 

9.6 

“0.159  -0.28240000 

9.8 

“0.156  -0.28970000 

10.0 

-0.153  -0.29680000 

10.2 

“0.150  -0.30360000 

10.4 

-0.147  -0.31020000 

*  0.15 


L  A  ft  B  0  A  3 

L AHBDA4 

5.35500000 

0.97656000 

7.1187000G 

1.14110000 

0.01103600 

0.291 18000 

0.02204800 

0.23325000 

0.02809000 

0.18552000 

0.03013800 

0.14551000 

0.02905300 

0. 1 1166000 

0.02564700 

0.00286400 

0.02064000 

0.05821600 

0.01461000 

0.03697200 

0.00600100 

0.01053900 

0.00458100 

0.01020000 

0.00121100 

0.00261200 

-0.00234500 

-0.00409600 

-0.00580800 

-0.01177700 

-0.01265200 

-0.02437200 

“0.01932000 

“0.03561700 

“0.02575800 

“0.04571000 

“0.03193700 

-0 . 05484200 

-0.03784400 

-0.06312700 

-0.04347600 

-0.07068500 

-0.04883900 

-0.07760900 

-0.05393900 

-0.00397900 

-0.05879000 

-0.08985900 

“0.06340200 

-0.09530600 

“0.06778900 

“0.10037000 

-0.07196400 

-0.10500000 

“0.07593900 

-0.10949000 

-0.07972700 

-0. 1 1362000 

-0.08333900 

-0. 11750000 

-0.08678600 

-0.121 1 4000 

-0.09007800 

“0.12450000 

-0.09322500 

-0.12783000 

-0.09623600 

-0.13090000 

-0.09911800 

“0.13301000 

-0.10188000 

-0.13657000 

-0.10453000 

-0.13920000 

-0. 10707000 

-0.14170000 

-0.10952000 

-0.14409000 

“0.11 186000 

-0.14636000 

-0.11412000 

-0.14054000 

Table  C . 1 


Lambda  Parameters 
( continued ) 


ALPHAS  «  0.90 


ALPHA4 

LAMB0A1 

LAHB0A2 

IAM80A3 

2.8 

0.739 

0.51740000 

4.41340000 

3.0 

0.739 

0.49070000 

6.39070000 

3.2 

0.778 

0.46610000 

8.21710000 

3.4 

-1.085 

0.17510000 

0.01327300 

3.6 

-0.933 

0.15860000 

0.02178400 

3.6 

-0.814 

0.13970000 

0.02601600 

4.0 

-0.718 

0.11920000 

0.02686100 

4.2 

-0.639 

0.09790000 

0.02509600 

4.4 

-0.575 

0.07620000 

0.02142400 

4.6 

-0.521 

0.05470000 

0.01644400 

4.8 

-0.477 

0.03370000 

0.01063300 

5.0 

-0.439 

0.01320000 

0.00432800 

5.1 

-0.422 

0.00333900 

0.00111100 

5.2 

-0.407 

-0.00640000 

-0.00215400 

5.3 

-0.394 

-0.01590000 

-0.00542800 

5.4 

-0.379 

-0.02520000 

-0.00869400 

5.6 

-0.356 

-0.04320000 

-0.01515300 

5.8 

-0.335 

-0.06040000 

-0.02145400 

6.0 

-0.317 

-0.07680000 

-0.02755000 

6.2 

-0.301 

-0.09250000 

-0.03341500 

6.4 

-0.286 

-0.10740000 

-0.03903500 

6.6 

-0.274 

-0.12160000 

-0.04440700 

6.6 

-0.262 

-0.13520000 

-0.04953400 

7.0 

-0.252 

-0.14820000 

-0.05442200 

7.2 

-0.242 

-0.16050000 

-0.05908100 

7.4 

-0.233 

-0.  17240000 

-0.06352000 

7.6 

-0.226 

-0.18380000 

-0.06775200 

7.8 

-0.218 

-0.19460000 

-0.07178600 

8.0 

-0.212 

-0.20510000 

-0.07563400 

8.2 

-0.205 

-0.21510000 

-0.07930600 

8.4 

-0.200 

-0.22470000 

-0.08281400 

8.6 

-0.194 

-0.23390000 

-0.08616600 

8.8 

-0.189 

-0.24290000 

-0.08937300 

9.0 

-0.184 

-0.25140000 

-0. 09244200 

9.2 

-0.180 

-0.25970000 

-0.09538100 

9.4 

-0.176 

-0.26760000 

-0.09819900 

9.6 

-0.172 

-0.27530000 

-0.10090000 

9.8 

-0.168 

-0.28280000 

-0.10350000 

10.0 

-0.165 

-0.28990000 

-0.10599000 

10.2 

-0.162 

-0.29690000 

-0.10839000 

10.4 

-0.159 

-0.30360000 

-0.11070000 

Tabl 


LAMBSA4  ALPHA4 

0 . 76325000  3.0 

1.00870000  3.2 

1.12200000  3.8 

0.25483000  4.0 

O.283B90U0  4.2 

0.161 53000  4.4 

0.12577000  4.6 

0.09533700  4.8 

0 . 06926600  5.0 

0.04679100  5.2 

0.02729300  5.4 

0.01020000  5.6 

0.00252600  5.8 

-0 . 00473500  6.0 

-0.01160000  6.2 

-0.01801300  6.4 

-0.02984800  6.6 

-0.04046200  6.8 

-0.05003400  7.0 

-0.05871100  7.2 

-0.06661500  7.4 

-0.07384500  7.6 

-0.00048600  7.8 

-0.08660900  8.0 

-0.09227300  8.2 

-0.09753000  8.4 

-0.10242000  8.6 

-0.10699000  8.8 

-0.11126000  9.0 

-0.11527000  9.2 

-0.11904000  9.4 

-0.12259000  9.6 

-0.12594000  9.8 

-0.12910000  10.0 

-0.13210000  10.2 

0.13494000  10.4 

0.13765000  10.6 

0.14022000  10.8 

0.14267000  11.0 

0.14500000 
0.14723000 


ALPHA3 


LAN80A1 

LAM80A2 

0.773 

0.50060000 

0.796 

0.47570000 

-0.906 

0.14550000 

-0.797 

0.12640000 

-0.709 

0.10620000 

-0.637 

0.08530000 

-0.576 

0.06440000 

-0.526 

0.04370000 

-0.484 

0.02350000 

-0.448 

0.00400000 

-0.417 

-0.01460000 

-0.390 

-0.03290000 

-0.367 

-0.05020000 

-0.346 

-0.06670000 

-0.328 

-0.08250000 

-0.312 

-0.09760000 

-0.298 

-0.11200000 

-0.285 

-0.12570000 

-0.273 

-0.13890000 

-0.263 

-0.15140000 

-0.253 

-0.16350000 

-0.245 

-0.17500000 

-0.236 

-0.18600000 

-0.229 

-0.19660000 

-0.222 

-0.20680000 

-0.216 

-0.21650000 

-0.210 

-0.22590000 

-0.204 

-0.23500000 

-0.199 

-0.24370000 

-0.194 

-0.25210000 

-0.190 

-0.26010000 

-0.186 

-0.26790000 

-0.182 

-0.27550000 

-0.178 

-0.28280000 

-0.174 

-0.28980000 

-0.171 

-0.29660000 

-0.168 

-0.30320000 

-0.165 

-0.30960000 

-0.162 

-0.31580000 

U  .  T  J  - 

LANB0A3 

LAHBDM 

5.52450000 

0.85032000 

7.40030000 

1.00890000 

0.02074900 

0.17941000 

0.02364800 

0.14140000 

0.02365600 

0.10913000 

0.02146700 

0.08147900 

0.01766500 

0.05765800 

0.01281200 

0.03698800 

0.00722600 

0.01888900 

0.00127600 

0.00307400 

-0.00487300 

-0.01095800 

-0.01105400 

-0.02344900 

-0.01716300 

-0.03463200 

-0.02313300 

-0.04470100 

-0.02892200 

-0.05381400 

-0.03450300 

-0.06210100 

-0.03986400 

-0.06967000 

-0.04500700 

-0.07661200 

-0.04991600 

-0.08300400 

-0.05461200 

-0.08890800 

-0.05909700 

-0.09438100 

-0.06338000 

-0.09947000 

-0.06747000 

-0. 10421000 

-0.07137600 

-0.10865000 

-0.07510800 

-0.11280000 

-0.07867600 

-0.11670000 

-0.08209900 

-0.12037000 

-0.08535500 

-0.12383000 

-0.08848400 

-0.12710000 

-0.09148200 

-0.13020000 

-0.09435700 

-0.13313000 

-0.09711600 

-0.13592000 

-0. 09976600 

-0.13856000 

-0.10231000 

-0.14108000 

-0.10476000 

-0.14348000 

-0.10712000 

-0.14578000 

-0.10939000 

-0.14797000 

-0.11157000 

-0.15006000 

-0.11368000 

-0.15207000 

C . 1  Lambda  Parameters 
( continued) 


AL  PH  A  4 
3.0 

3.2 

3.4 

3 . 6 

3.8 
4.0 

4.2 

4.4 

4.6 

4.8 
5.0 

5.2 

5.3 

5.4 

5.5 

5.6 

5.8 
6.0 
6.2 

6.4 

6.6 

6.8 

7.0 

7.2 

7.4 

7.6 

7.8 

8.0 

8.2 

8.4 

8.6 

8.8 
9.0 
9.2 

9.4 
9.6 

9.8 

10.0 

10.2 

10.4 

10.6 

10.8 


AIPHA3  =  1  .00 


LAHBDA1 

L  Ah  8  0  A  2 

L Ah  B  DA  3 

0.855 

0.5060U000 

4.25260000 

0.822 

0.48570000 

6.49730000 

0.856 

0.46250000 

8.40460000 

0.908 

0.44150000 

10.48700000 

-1.010 

0.15090000 

0.01413100 

-0.887 

0.13330000 

0.01926500 

-0.787 

0.11420000 

0.02119300 

-0.705 

0.09430000 

0.02063600 

-0.638 

0.07410000 

0.01820300 

-0.5B1 

0.05390000 

0.01441600 

-0.533 

0.03400000 

0.00969500 

-0.493 

0.01470000 

0.00438300 

-0.474 

0.00519200 

0.00158400 

-0.445 

-0.00031700 

-0.00010100 

-0.442 

-0.01320000 

-0.00417600 

-0.428 

-0.02220000 

-0.00709700 

-0.401 

-0.03960000 

-0.01294800 

-0.379 

-0.05620000 

-0.01873800 

-0.358 

-0.07210000 

-0.02440700 

-0.340 

-0.08730000 

-0.02991600 

-0.324 

-0.10190000 

-0.03524000 

-0.310 

-0.11 580U00 

-0.04036700 

-0.297 

-0.12910000 

-0.04529100 

-0.285 

-0.14190000 

-0.05001200 

-0.275 

-0.15410000 

-0.05453400 

-0.265 

-0.16580000 

-0.05886100 

-0.256 

-0.17700000 

-0.06300000 

-0.248 

-0.18770000 

-0.06696100 

-0.240 

-0.19800000 

-0.07075000 

-0.233 

-0.20800000 

-0.07437700 

-0.227 

-0.21750000 

-0.07785000 

-0.220 

-0.22670000 

-0.08117600 

-0.215 

-0.23550000 

-0.08436400 

-0.209 

-0.24400000 

-0.08742100 

-0.205 

-0.25220000 

-0.09035400 

-0.200 

-0.26020000 

-0.09317100 

-0.195 

-0.26780000 

-0.09587600 

-0.191 

-0.27520000 

-0.09847700 

-0.187 

-0.28240000 

-0.10098000 

-0.184 

-0.28930000 

-0.10339000 

-0.180 

-0.29600000 

-0.10571000 

-0.177 

-0.30250000 

-0.10794000 

* 


AIPHA3  *1.10 


LAABBA4 

AIPHA4 

L  Ah  8  0 A 1 

L  Ah  8  0  A  2 

L AHI0A3 

IAIB6A4 

0.60467000 

3.2 

1.700 

0.37720000 

2.93560000 

0.11710000 

0.17809000 

3.4 

0.909 

0.48140000 

6.32360000 

0.74100000 

0.99148000 

3.6 

0.931 

0.46040000 

8.31040000 

0.86594000 

1.04910000 

3.8 

0.977 

0.44030000 

10.39100000 

0.93066000 

0.19962000 

4.0 

-1.108 

0.14580000 

0.00603500 

0.20127000 

0.15884000 

4.2 

-0.974 

0.12940000 

0.01250900 

0  .  16073000 

0.12441000 

4.4 

-0.868 

0.11170000 

0.01574300 

0.12675000 

0 .09500500 

4.6 

-0.781 

0.09320000 

0.01649700 

0.09774700 

0.06967500 

4.8 

-0.709 

0.07430000 

0.01535100 

0.07271000 

0.04771300 

5.0 

-0.648 

0. 05520000 

0.01275500 

0.05078200 

0.02853900 

5.2 

-0.596 

0.03650000 

0.00916800 

0.03183800 

0.01171900 

5.4 

-0.552 

0.01810000 

0.00483900 

0.01501700 

0.00406100 

5.5 

-0.532 

0.00903800 

0.00248400 

0.00734200 

-0.00024200 

5.6 

-0.517 

0.00099700 

0.00027900 

0.00079500 

-0.00994600 

5.7 

-0.497 

-0.00862900 

-0.00247900 

-0.00672600 

-0.01636500 

5.8 

-0.481 

-0.01720000 

-0.00504600 

-0.01318600 

-0.02817900 

6.0 

-0.452 

-0.03400000 

-0.01027100 

-0.02509700 

-0.03880000 

6.2 

-0.426 

-0.05010000 

-0.01553500 

-0.03582500 

-0.04839600 

6.4 

-0.404 

-0.06560000 

-0.02076200 

-0.04553400 

-0.05710900 

6.6 

-0.384 

-0.08050000 

-0.02590300 

-0.05436000 

-0.06505400 

6.8 

-0.366 

-0.09470000 

-0.03092100 

-0.06241700 

-0.07233100 

7.0 

-0.350 

-0. 1 0840000 

-0.03579500 

-0.06980200 

-0.07902000 

7.2 

-0.335 

-0.12150000 

-0.04051100 

-0.07659500 

-0.08519100 

7.4 

-0.322 

-0.13400000 

-0.04506100 

-0.08286600 

-0.09090400 

7.6 

-0.310 

-0.14600000 

-0.04944300 

-0.08867300 

-0.09620800 

7.8 

-0.299 

-0.15760000 

-0.05365700 

-0.09406600 

-0. 10115000 

8.0 

-0.289 

-0.16870000 

-0.05770600 

-0.099088UQ 

-0.10576000 

8.2 

-0.280 

-0.17930000 

-0.06159400 

-0.10378000 

-0.11007000 

8.4 

-0.271 

-0.18960000 

-0.06532700 

-0.10817000 

-0.11412000 

8.6 

-0.263 

-0.19940000 

-0.06891 100 

-0.11229000 

-0.11793000 

8.8 

-0.256 

-0.20890000 

-0.07235200 

-0. 1 1616000 

-8.12151000 

9.0 

-0.249 

-0.21300000 

-0.07565600 

-0.11981000 

-0.12490000 

9.2 

-0.242 

-0.22680000 

-0.07883000 

-0.12325000 

-0.12810000 

9.4 

-0.236 

-0.23530000 

-0.08188000 

-0.12650000 

-0.13113000 

9.6 

-0.231 

-0.24350000 

-0.08481200 

-0.12959000 

-0.13400000 

9.8 

-0.226 

-0.25140000 

-0.08763200 

-0.13251000 

-0.13673000 

10.0 

-0.221 

-0.25900000 

-0.09034600 

-0.13529000 

-0.13933000 

10.2 

-0.216 

-0.26640000 

-0.09295800 

-0.13793000 

-0.14180000 

10.4 

-0.211 

-0.27360000 

-0.09547500 

-0.14045000 

-0.14417000 

10.6 

-0.207 

-0.28050000 

-0.09790000 

-0.14285000 

-0.14642000 

10.8 

-0.203 

-0.28720000 

-0.10024000 

-0.14514000 

-0.14858000 

11.0 

-0.200 

-0.29370000 

-0.10249000 

-0.14733000 

e  C.l  Lambda  Parameters 
( continued ) 


AIFHA3  ;  1.20 


ALPHA4 

L AH0OA1 

L AH8BA2 

L  Ah8  DA  3 

3.2 

2.314 

0.31240000 

3.00060000 

3.4 

2.116 

0.33370000 

3.26790000 

3.6 

1.019 

0.47530000 

5.79000000 

3.8 

1.009 

0.45880000 

7.93240000 

4.0 

1.044 

0.44010000 

9.99990000 

4.2 

>0.563 

-0.00020000 

-0.00004400 

4.4 

-1.083 

0.12780000 

0.00509800 

4.6 

-0.965 

0.11 140000 

0.00996B00 

4.8 

-0.070 

0.09410000 

0.01220800 

5.0 

-0.791 

0.07640000 

0.01244600 

5.2 

-0.724 

0.05060000 

0.01115300 

5.4 

“0.660 

0.04000000 

0.00870500 

5.6 

-0.619 

0.02330000 

0.00541100 

5.7 

-0.597 

0.04160000 

0.00352500 

5.8 

-0.577 

0.00610000 

0.00151 500 

5.9 

-0.550 

-0.00231900 

-0.00059400 

6.0 

-0.562 

-0.00096200 

-0.00024500 

6.2 

-0.500 

-0.02600000 

-0.00734300 

6.4 

-0.400 

-0.04240000 

-0.01204000 

6.6 

-0.455 

-0.05750000 

-0.01679100 

6.8 

-0.432 

-0.07200000 

-0.02153300 

7.0 

-0.412 

-0.08600000 

-0.02621900 

7.2 

-0.394 

-0.09940000 

-0.03001800 

7.4 

-0.377 

-0.11230000 

-0.03530600 

7.6 

-0.363 

-0.12470000 

-0.03966900 

7.0 

-0.349 

-0.13660000 

-0.04309900 

8.0 

-0.337 

-0.14800000 

-0.04798900 

8.2 

-0.325 

-0.15900000 

-0.05194000 

8.4 

-0.315 

-0.16960000 

-0.05575100 

0.6 

-0.305 

-0.17970000 

-0.05942400 

8.8 

-0.296 

-0.18950000 

-0.06296200 

9.0 

-0.288 

-0.19900000 

-0.06637000 

9.2 

-0.280 

-0.20810000 

-0.06965200 

9.4 

-0.273 

-0.21600000 

-0.07201200 

9.6 

-0.266 

-0.22530000 

-0.07505600 

9.0 

-0.259 

-0.2335OOOO 

“0.07878900 

10.0 

-0.253 

-0.24140000 

-0.08161500 

10.2 

-0.248 

-0.24900000 

-0.08433900 

10.4 

-0.243 

-0.25640000 

-0.08696500 

10.6 

-0.237 

-0.26360000 

-0.00949900 

10.8 

-0.233 

-0.27050000 

-0.09194400 

1 1.0 

-0.228 

-0.27720000 

-0.09430500 

11.2 

-0.224 

-0.20370000 

-0.09660000 

11.4 

-0.220 

-0.29010000 

-0.09800000 

Tabl 


ALPHAJ  «  1.30 


LAMBliM 

ALPHA4 

LAM8DA1 

L AM80A2 

LAMBDA3 

LA8BIA4 

0.03356100 

3.8 

1.311 

0.44150000 

4.39930000 

0.30896000 

0 ,04339800 

4.0 

1.102 

0.45600000 

7.24220000 

0.60354000 

0.58346000 

4.2 

1.114 

0.44010000 

9.33050000 

0.70343000 

0.7399200U 

4.4 

1.155 

0.42360000 

11.51700000 

0.75751000 

0.11738000 

4.6 

1.210 

0.40760000 

13.96100000 

0.78535000 

-0.00014200 

4.8 

-1.004 

0.11290000 

0.00317400 

0.14353000 

0.16750000 

5.0 

-0.975 

0.09600000 

0.00722500 

0.11303000 

0.13292000 

5.2 

-0.886 

0. 0802000U 

0.00903500 

0.08702600 

0.10364000 

5.4 

-0.812 

0.06340000 

0.00914800 

0.06448100 

0.07841500 

5.6 

-0.749 

0.04660000 

0.00795900 

0.04471000 

0.05645700 

5.8 

-0.695 

0.02990000 

0.00578300 

0.02727800 

0.03719000 

6.0 

-0.604 

0.00028600 

0.00006619 

0.00023900 

0.02017100 

6. 1 

-0.617 

0.00044600 

0.00010000 

0.00037500 

0.01240000 

6.2 

-0.616 

-0.00052600 

-0.00011000 

-0.00044200 

0.00505000 

6.3 

-0.589 

-0.01040000 

-0.00245000 

-0.00850400 

-0.00188400 

6.4 

-0.571 

-0.01820000 

-0.00439900 

-0.01457400 

-0.00078400 

6.6 

-0.539 

-0.03330000 

-0.00846900 

-0.02583000 

-0.02058100 

6.8 

-0.51 1 

-0.04800000 

-0.01268000 

-0.03603000 

-0.03151900 

7.0 

-0.486 

-0.06220000 

-0.01690100 

-0.04533200 

-0.04142900 

7.2 

-0.463 

-0.07590000 

-0.02129100 

-0.05382900 

-0.05044800 

7.4 

-0.443 

-0.00900000 

-0.02557600 

-0.06162400 

-0.05868800 

7.6 

-0.424 

-0.10100000 

-0.02980400 

-0.06079900 

-0.06624500 

7.8 

-0.407 

-0. 11400000 

-0.03395200 

-0.07542600 

-0.07319900 

8.0 

-0.392 

“0.12570000 

-0.03800600 

-0.08156400 

-0.07962000 

0.2 

-0.370 

-0.13710000 

-0.04195300 

-0.08726500 

-0.08556800 

8.4 

-0.365 

“0.14800000 

-0.04570800 

-0.09257600 

-0.09109300 

8.6 

-0.353 

-0.15850000 

-0.04950700 

-0.09753400 

-0.09623900 

8.0 

-0.342 

-0.16060000 

-0.05310800 

-0.10217000 

-0.10104000 

9.0 

-0.332 

-0.17840000 

-0.05659200 

-0.10653000 

-0.10554000 

9.2 

-0.323 

-0.18700000 

-0.05995900 

-0.11062000 

-0.10976000 

9.4 

-0.314 

-0.19690000 

-0.06321300 

-0.11447000 

-0.11373000 

9.6 

-0.306 

-0.20570000 

-0.06635500 

-0.1 1810000 

-0.11746000 

9.8 

-0.298 

-0.21420000 

-0.06939000 

-0.12154000 

-0.12099000 

10.0 

-0.291 

-0.22240000 

-0.07232100 

-0. 12479UOO 

-0.12432000 

10.2 

-0.204 

-0.23030000 

-0.07515200 

-0. 12787000 

-0.12748000 

10.4 

-0.277 

-0.23790000 

-0.07788600 

-0.13080000 

-0.13047000 

10.6 

-0.271 

-0.24540000 

-0.08052700 

-0.13358000 

“0.13331000 

10.0 

-0.266 

-0.25250000 

-0.08307900 

-0.13624000 

-0.13602000 

1  1  .0 

-0.260 

-0.25950000 

-0.08554600 

-0.13876000 

-0.13859000 

11.2 

-0.255 

-0.26620000 

-0.0e793100 

-0.14117000 

-0.14105000 

11.4 

-0.250 

-0.27200000 

-0.09023800 

-0.14348000 

-0.14339000 

11.6 

-0.246 

-0.27910000 

-0.09247000 

-0.14568000 

-0.14560000 

11.0 

-0.242 

-0.28530000 

-0.09463000 

-0.14780000 

o.n7euooo 

C . 1  Lambda  Parameters 
( continued) 


AL  PN A3 


ALPHA4 

LAA8DA1  LAH80A2 

4.2 

1.255  0.44530000 

4.4 

1.200  0.43880000 

4.6 

1.219  0.42460000 

4.8 

1.262  0.40980000 

5.0 

1.318  0.39530000 

5.2 

-1.106  0.10110000 

5.4 

-1.001  0.08550000 

5.6 

-0.916  0.06970000 

5.8 

-0.844  0.05380000 

6.0 

-0.782  0.03790000 

6.2 

-0.729  0.02220000 

6.3 

-0.706  0.01450000 

6.4 

-0.683  0.00680000 

6.5 

-0.660  -0.00122600 

6.6 

-0.643  -0.00830000 

6.8 

-0.607  -0.023U0O00 

7.0 

-0.575  -0.03730000 

7.2 

-0.546  -0.05110000 

7.4 

-0.521  -0.06450000 

7.6 

-0.497  -0.07750000 

7.8 

-0.476  -0.09000000 

8.0 

-0.457  -0.10210000 

8.2 

-0.440  -0.11370000 

8.4 

-0.424  -0.12500000 

8.6 

-0.409  -0.13580000 

8.8 

-0.396  -0.14630000 

9.0 

-0.383  -0.15640000 

9.2 

-0.372  -0.16610000 

9.4 

-0.361  -0.17560000 

9.6 

-0.351  -0.18470000 

9.8 

-0.342  -0.19340000 

10.0 

-0.333  -0.20190000 

10.2 

-0.325  -0.21010000 

10.4 

-0.317  -0.21810000 

10.6 

-0.310  -0.22580000 

10.8 

-0.303  -0.23320000 

11.0 

-0.297  -0.24050000 

11.2 

-0.291  -0.24750000 

11.4 

-0.285  -0.25420000 

11.6 

-0.279  -0.26080000 

11.8 

-0.274  -0.26720000 

12.0 

-0.269  -0.27340000 

12.2 

-0.265  -0.27940000 

*1.40 


l AM8DA3 

L  AflBDM 

6.10130000 

0.42931000 

8.40330000 

0.58043000 

10.53200000 

0.65450000 

12.79900000 

0.69618000 

15.34800000 

0.71771000 

0.00078700 

0. 12683000 

0.00454600 

0. 01910100 

0.00629600 

0.07537100 

0.00653000 

0.05472000 

0.00560300 

0.03654100 

0.00378500 

0.02040500 

0.00261  100 

0.01300000 

0.00129200 

0.00598700 

-0.00024400 

-0.00105200 

-0.00170200 

-0.00696800 

-0.00506000 

-0.01866600 

-0.00867000 

-0.02927700 

-0.01244600 

-0.03894 1 U0 

-0.01631600 

-0.04777600 

-0.02023400 

-0.05588200 

-0.02415300 

-0.06334400 

-0.02804500 

-0.07023400 

-0.03188500 

-0.07661600 

-0.03565700 

-0.08254300 

-0.03934900 

-0.08806300 

-0.04295300 

-0.09321400 

-0.04646200 

-0.09803400 

-0.04987400 

-0.10255000 

-0.05318700 

-0.10680000 

-0.05640000 

-0. 11080000 

-0.05951500 

-0.1 1457000 

-0.06253300 

-0. 11813000 

-0.06545500 

-0. 121  50000 

-0.06828500 

-0. 12470000 

-0.07102500 

-0.12773000 

-0.07367800 

-0.13061000 

-0.07624600 

-0. 13336000 

-0.07873300 

-0.13597000 

-0.081 14100 

-0. 13847000 

-0.08347400 

-0.14085000 

-0.08573400 

-0. 14313UOO 

-0.08790000 

-0. 14530000 

-0.09000000 

-0.  14740000 

ALPHA3 


ALPHA4 

tAHBDAI  t  AH  8  DA  2 

4.4 

1.966  0.35820000 

4.6 

1.337  0.43110000 

4.8 

1.302  0.42390000 

5.0 

1  .323  0.41  U0000 

5.2 

1.365  0.39810000 

5.4 

1.419  0.38500000 

5 . 6 

1.484  0.37210000 

5.8 

-1.042  0.07730000 

6.0 

-0.957  0.06220000 

6.2 

-0.885  0.04710000 

6.4 

-0.824  0.03210000 

6,6 

-0.688  0.00056600 

6.7 

-0.747  0.00996200 

6.8 

-0.714  -0.00029000 

6.9 

-0.704  -0.00444600 

7.0 

-0.684  -0.01150000 

7.2 

-0.647  -0.02540000 

7.4 

-0.615  -0.03900000 

7.6 

-0.585  -0.05210000 

7.8 

-0.559  -0.06480000 

8.0 

-0.535  -0.07720000 

8.2 

-0.513  -0.08910000 

8.4 

-0.494  -0.10070000 

8.6 

-0.475  -0.11180000 

8.8 

-0.459  -0.12260000 

9.0 

-0.443  -0.13300000 

9.2 

-0.429  -0  14310000 

9.4 

-0.416  -0.15280000 

9.6 

-0.404  -0.16220000 

9.8 

-0.392  -0.17140000 

10.0 

-0.382  -0.18020000 

10.2 

-0.372  -0.18870000 

10.4 

-0.363  -0.19690000 

10.6 

-0.354  -0.20490000 

10.8 

-0.346  -0.21270000 

11.0 

-0.338  -0.22010000 

11.2 

-0.331  -0.22740000 

11.4 

-0.324  -0.23450000 

11.6 

-0.317  -0.24130000 

11.8 

-0.311  -0.24790000 

12.0 

-0.305  -0.25440000 

12.2 

-0.300  -0.26070000 

12.4 

-0.294  -0.26670000 

12.6 

-0.289  -0.27270000 

12.8 

-0.284  -0.27840000 

Table  C.l  Lambda  Parameters 
( continued ) 


--  1.50  . 

L  AHB&A3 
4.50420000 
7.  10470000 
9.32070000 
1 1.48100000 
13.79600000 
1  6 .38600000 
19.36300000 
0.00194900 
0.00390700 
0.00444100 
0.00388500 
0.00010400 
0.00153800 
•0.00004897 
*0.00076800 
*0.00208800 
-0.00498900 
-0.00815600 
-0.01150900 
-0.01498300 
-0.01852800 
-0.02210500 
-0.02568100 
-0.02923300 
-0.03274300 
-0.03620000 
*0.03960000 
-0.04289900 
-0.04613400 
-0.04928800 
-0.05235900 
-0.05534500 
-0.05824700 
-0.06106700 
-0.06380400 
-0.06646000 
-0.06903900 
-0.07154100 
-0.07396800 
-0.07632400 
-0.07861100 
-0.08083000 
-0.08298400 
-0.08507600 
-0.08710700 


LAHBBA4 
0.12876000 
0.42910000 
0.54132000 
0.60187000 
0.63706000 
0.65586000 
0.66279000 
0.08991400 
0.04770300 
0.04834600 
0.03126600 
0.00049400 
0.00905900 
-0.00025600 
-0.00388200 
-0.00987500 
-0.02101900 
-0.031 16200 
-0.04043100 
-0.04893200 
-0.05675500 
-0.06397600 
-0.07066200 
-0.07486900 
-0.08264600 
-0.08800000 
-0.09310000 
-0.09780300 
-0. 1 0224000 
-0. 10642000 
-0.11035000 
-0.11407000 
-0.11759000 
-0.12092000 
-0.12408000 
-0.12708000 
-0.12994000 
-0. 13266000 
-0. 13526000 
-0. 13774000 
-0.14011000 
-0.14238000 
-0.14455000 
-0. 14663000 
-0.14863000 


ALPHA4 

LAtimi 

5.6 

1.313 

4.0 

1.377 

4.2 

1.447 

4.4 

-1.011 

4.4 

-0.937 

4.6 

-0.873 

7.0 

-0.744 

7.1 

-0.794 

7.2 

-0.771 

7.3 

-0.731 

7.4 

-0.731 

7.4 

-0.493 

7.8 

-0.439 

0.0 

-0.429 

8.2 

-0.402 

0.4 

-0.377 

0.4 

-0.334 

0.8 

-0.333 

9.0 

-0.314 

9.2 

-0.497 

9.4 

-0.480 

9.4 

-0.445 

9.8 

-0.431 

10.0 

-0.438 

10.2 

-0.424 

10.4 

-0.413 

10.4 

-0.404 

10.0 

-0.395 

11.0 

-0.305 

11.2 

-0.374 

11.4 

-0.348 

11.4 

-0.360 

11.0 

-0.353 

12.0 

-0.346 

12.2 

-0.339 

12.4 

-0.333 

12.4 

-0.327 

12.8 

-0.321 

13.0 

-0.313 

13.2 

-0.310 

13.4 

-0.303 

13.4 

-0.301 

13.8 

-0.294 

-  ALPHA] 

LAM0QA2 
0.37620000 
0.34440000 
0. 3S260000 
0.03730000 
0.04300000 
0.02870000 
‘0.00042200 
0.00777310 
‘0.00034100 
‘0.00392400 
‘0.01240000 
-0.02380000 
-0.03870000 
-0.05120000 
‘0.04340000 
-0.07320000 
-0.08:40000 
-0.09770000 
-0.10840000 
-0.11880000 
-0. 12880000 
-0.13840000 
-0.14800000 
-0.15710000 
-0.14390000 
-0.17450000 
-0.18280000 
-0.19080000 
-0.19840000 
-0.20420000 
-0.21350000 
-0.22060000 
-0.22750000 
-0.23420000 
-0.24070000 
-0.24700000 
-0.25320000 
-0.23920000 
-0.24300000 
-0.27070000 
-0.27420000 
-0.26160000 
-0.28400000 


«  1.40  . 

LAM0QA3 

17.04300000 

19.91300000 

23.33400000 

0.00149900 

0.00248401 

0.00239700 

-0.00004334 

0.00094900 

-0.00004434 

-0.00015800 

-0.00194200 

-0.00*38300 

-0.00711101 

-0.01004900 

-0.01313400 

-0.01432200 

-0.01954700 

-0.02283900 

-0.02411401 

-0.02937100 

-0.03259400 

-0.03377400 

-0.03890200 

-0.04194900 

-0.04497000 

-0.04790400 

-0.05074700 

-0.05355900 

-0.05427900 

-0.03092800 

-0.04130600 

-0.06401400 

-0.06645400 

-0.06882700 

-0.07113400 

-0.07337700 

-0.07333900 

-0.07748000 

-0.07974300 

-0.08173000 

-0.08370208 

-0.08360200 

-0.06745000 


Table  C.l 


t 


LAI80A4 
0.39931000 
0.40731000 
0.40744000 
1.06343300 
0.04484700 
0.07649100 
O.OOO370OO 
0.00717708 
-0.00030900 
-0. SO52790O 
-0.01105900 
-0. 02  T  8  24  00 
-0.03164700 
-0.14044400 
-0.04092000 
-0.03433100 
-0.06361100 
-0.07014000 
-0.07425300 
-0.08193400 
-0.00724400 
-0.09221 700 
-0.09480500 
-0. 10128000 
-0.10541000 
-0.10932000 
-0  . 1 1301000 
-0.11 430000 
-0.11982000 
•0.12296000 
-0.12594000 
*0. 1260UQOO 
-0.13152008 
-0.1341 1000 
-0.13459000 
-0.13094000 
0.14123000 
0.14340000 
0.14549000 
0.14749000 
0. 1 4942000 
0. 151 77000 
0.15305000 


ALPHA] 


ALPHA4 

LAA0OA1 

LAfllOA? 

4.8 

4.734 

0.14100000 

5.0 

2.792 

0.28500000 

3.2 

1.919 

0.36380000 

5.4 

1.342 

0.40140000 

3.4 

1.514 

0.39830000 

3.8 

1.526 

0 .38940000 

4.0 

1.560 

0.3791Q000 

4.2 

1.407 

0.34850000 

4.4 

1.444 

0.35^80000 

4.4 

1.729 

0.34720000 

4.0 

1.002 

0.33470000 

7.0 

-1.001 

0.041 10000 

7.2 

*0.933 

0.02750000 

7.4 

-0.878 

0.01420000 

7.3 

-0.832 

0.00734400 

7.4 

-0.024 

-0.00020000 

7.7 

-0.806 

-0.00346900 

7.8 

-0.784 

-0.01190000 

8.0 

-0.743 

-0.02430000 

8.2 

-0.710 

-0.03670000 

8.4 

-0.478 

-0. 04870000 

8.4 

-0.449 

-0.04030000 

0.1 

-0.423 

-0.07170000 

9.0 

-0.399 

0.08260000 

9.2 

-0.377 

0.09330000 

9.4 

-0.337 

-0.10360000 

9.4 

-0.330 

0.11370000 

9.8 

-0.321 

-0.12340000 

10.0 

-0.305 

-0.1328U000 

10.2 

-0.490 

-0.14200000 

10.4 

0.476 

-0.15080000 

10.6 

-0.463 

-0.15940000 

10.8 

-0.431 

-0.14760000 

11.0 

-0.440 

-0. 17590000 

1 1 .2 

-0.429 

-0.18370000 

11.4 

-0.419 

'0.19130000 

11.4 

-0.409 

-0.19870000 

11.8 

-0.401 

-0.20590000 

12.0 

-0.392 

-0.21290000 

12.2 

-0.384 

0.21940000 

12.4 

-0.374 

-0.22620000 

12.4 

-0.349 

-0.23240000 

12.8 

-0.342 

-0.23880000 

13.0 

-0.354 

-0.24500000 

13.2 

-0.350 

-0.25080000 

13.4 

-0.344 

-0.25660000 

13.4 

-0.338 

0.24220000 

13  8 

-0.333 

-0.24750000 

‘1.70 


LAH80A3 

L4«fclA4 

3.53430000 

-0.09290100 

5.14110000 

0.041 2530u 

3.86250000 

0.17943000 

0.24260000 

0.34039000 

10.42100000 

0.44444000 

12.38400000 

0.49487000 

14.06200000 

0.52822000 

17.39700000 

0.54705000 

20.21100000 

0.55600000 

23.40800000 

0.55959000 

27.08800000 

0.5S484O0O 

0.00102700 

0.I44OS4OO 

0.00131300 

0.02802300 

0.001 14200 

0.01375400 

0.00049600 

0.01717900 

-0.00002600 

-O.OIO232O0 

-0.00041900 

-0.00500000 

-0.00146300 

-0.01070000 

-0.00342300 

-0.021 19900 

-O.U0370300 

-O.O3O031OO 

-0.00022500 

-0.03947300 

-0.01092400 

-0.04781400 

-0.01375100 

-0.05533500 

-0.01666600 

•0.04230400 

-0.01963700 

*0.04676000 

*0.02243600 

•0.07401300 

-0.02544300 

-0.08044400 

-0.02844000 

-0.08571800 

-0.03141400 

•O.O9O6oJO0 

-0.03435400 

-0.09530900 

-0.03743300 

-0.09960400 

-0.04030400 

-0.10381000 

-0.04310100 

-0.10771J0O 

-0.04584300 

-O.M140DOD 

-0.04052400 

-0. 1 1409000 

-0.05114800 

-0. 1 1821000 

-0.05371000 

-0.12136000 

-0.05421 100 

-0.12436000 

-0 .0584  5000 

-0.12722000 

-0.04102800 

-0.12994000 

-0.06334700 

-0.13255000 

-0.04540704 

-0.13504000 

-0.04780000 

-0.13740000 

-0.07000000 

-0.13970000 

-0.07200000 

0. 14190000 

-0.07410000 

-0.14400000 

-0.07410000 

-0.14400000 

-0.07800000 

-0.14790000 

Lambda  Parameters 
( continued) 


71*  L 


— 

-  ALPIA3 

ALPHAS 

LAAICA1 

IAAIIA2 

5.4 

2.841 

0.27400000 

5. A 

2.040 

0.34810000 

5.8 

1.702 

0.38560000 

6.0 

1.632 

0.38630000 

6.2 

1.631 

0.37470000 

6.4 

1.656 

0.37040000 

6.6 

1.647 

0.36150000 

6.1 

1.748 

0.35140000 

7.0 

1.807 

0.34230000 

7.2 

1.874 

0.33270000 

7.4 

-1.034 

0.03670000 

7.6 

-1.007 

0.02830000 

7.1 

-0.446 

0.01550000 

7.f 

‘0.418 

0.00417700 

1.0 

-0.000 

-0.00040000 

8. 1 

-0.060 

-0.00324100 

8.2 

-0.846 

-0.00440000 

8.4 

-0.004 

-0.02150000 

1.6 

-0.766 

-0.03330000 

8.1 

-0.733 

-0.04400000 

4.0 

-0.702 

-o. 05540000 

4.2 

-0.674 

-0.06660000 

4.4 

-0.644 

-0.07740000 

4.6 

-0.625 

-0.08770000 

4.8 

-0.604 

-0.04770000 

10. 0 

-0.584 

-0. 10740000 

10.2 

-0.565 

-0.11640000 

10. 4 

-0.548 

*0.12600000 

10.6 

-0.532 

-0.1 J44QOOO 

10. 8 

-0.317 

0.14360000 

11.0 

-0.503 

-0.15200000 

11.2 

-0.440 

-o.  UUI0000 

11.4 

-0.478 

-0.16000000 

11.6 

-0.466 

-0.17570000 

11.0 

-0.456 

-0.10320000 

12.0 

-0.445 

-0.14040000 

12.2 

-0.436 

-0. 14750000 

12.4 

-0.427 

-0.20430000 

12.6 

-0.410 

*0.21100000 

12.1 

-0.410 

-0.21750000 

13.0 

-0. 402 

-0.22380000 

13.2 

-0.344 

-0.22440000 

13.4 

-0.308 

-0.23540000 

13.6 

-0.381 

-0.24170000 

13.8 

-0.374 

-0.24*30000 

14.0 

-0.360 

-0.25300000 

14.2 

-0.362 

0.25830000 

14.4 

-0.357 

-0.26320000 

«  1  .  IU 


LAAI5A3 

L AI0IA4 

5.60740001 

0.04143800 

6.28120000 

0.13650000 

8.44740000 

0.31116000 

10.60400000 

0.39551000 

12.75000000 

8.44558000 

1 4.44500000 

1.47764080 

17.42400000 

0.49797000 

20.10400000 

0,58988800 

23. 10100001 

0.51539008 

26.44200000 

0.51514800 

0.0 

8.03460100 

0.00037800 

0.02978701 

0.00064600 

0.01555000 

0.00044000 

0.00900600 

0.00002600 

-1.01034000 

-0.00025400 

-0.00310200 

-0.00002600 

-0.00872100 

-0.00226900 

-0.01919300 

-0.00410300 

-0.02176300 

-0.00614000 

-0 . 03755200 

-0.00848900 

-0.04565000 

-0.01044800 

-0.05313900 

-U. 01352600 

-  0.06008 50C 

-0.01 610000 

-0.06654600 

-0.01640601 

-0.07257200 

-0.02165600 

-0.07820400 

*0.02442100 

-0.68348000 

-0. 027 1 6500 

-0.80043300 

-0.02443600 

-0.0VJ09I00 

0.03266300 

-o.orMeooo 

-0.03535400 

-0.10162000 

-0.03801800 

-0.10554100 

-0.04063400 

-0.10423000 

-0.04320600 

0. 1  12^7000 

-0.04572600 

-0. 1 1611000 

-0.04820100 

-0.11928000 

-0.05062201 

-0.12231000 

-0.05244100 

-0.12519000 

-0.05530700 

0.12794000 

-0.05757000 

-0.13057000 

-0.05470100 

-0.13301000 

-0.0614410 0 

0. 1 J549O00 

-0.06400000 

0.13700000 

-U. 06610000 

0.14000000 

-0.06610000 

0.14210000 

-0.07013000 

0.14420000 

-o.o^ocooo 

-0.14410000 

-0.07370000 

0.14  7f0000 

ALPHA] 


ALPHA4 

UA80A1 

LAAI0A2 

5.0 

3.084 

0.26760000 

6.0 

2.333 

0.32510000 

6.2 

1.876 

0.36750000 

6.4 

1.761 

0.37380000 

6.6 

1.741 

0.36940000 

6.0 

1.755 

0.36300000 

7.0 

1.707 

0.35440000 

7.2 

1.131 

0.34640000 

7.4 

1  .103 

0.33770000 

0.0 

-1.023 

0.02200000 

8.2 

-1.011 

0.01750000 

0.4 

-0.468 

0. 00644100 

8.5 

-0.446 

0.00123400 

0.6 

-0.417 

-0.00544600 

8.7 

-0.843 

-0.011 30000 

8.0 

*0.871 

-0.01710000 

4.0 

0.631 

-0.02860000 

4.2 

-0.744 

-0.03450000 

4.4 

-0.761 

-0.05030000 

4.6 

-0.731 

0.06040000 

4.6 

-0.704 

0.07110000 

10.0 

-0.678 

-0.08100000 

10.2 

-0.*55 

-0. 04000000 

10.4 

0.634 

-0.10020000 

10.6 

0.614 

-0.10460000 

10.6 

-0.545 

-0.11830000 

11.0 

0.578 

-0.12700000 

11.2 

-0.562 

-0.13340000 

11.4 

-0.547 

-0.14360000 

11.6 

-0.533 

~0. 15U0000 

1 1.8 

-0.320 

-0.15430000 

12.0 

-0.507 

-0.16640000 

12.2 

-0.445 

-0.17420000 

12.4 

0.464 

-0.18130000 

12.6 

-0.474 

-u. 16630000 

12.0 

-0.464 

-0.14500000 

13.0 

-0.455 

0.20160000 

13.2 

-0.446 

0.20800000 

13.4 

0.437 

-0 .21420000 

13.6 

-0.424 

-0.22030000 

13.8 

-0.421 

0.22620000 

I4.C 

0.416 

-0.23200000 

14.2 

-0.407 

-0.23760000 

14.4 

-0.600 

0.24310000 

14.6 

-0.346 

0.24660000 

14.8 

-0.388 

-0.25370000 

15.0 

-0.J62 

-0.25860000 

15.2 

-0.3*4 

0.2*370000 

15.4 

0.371 

0.26860000 

*  1  -  to 


LAHI0A3 

LA/11044 

6.05750000 

0.0334920 0 

6.34460000 

0.12351000 

0.44810000 

0.25763000 

10.37300000 

1.34266010 

12.67400000 

0. J945700C 

14.06000000 

0.42153000 

17.10800000 

0.45113000 

14.72100001 

0.46540800 

22.51400000 

0.47366000 

0.0 

1.12300000 

0.0 

fl  81011000 

0.00015000 

0.80643)00 

0.00004121 

0.01121500 

-0.00025700 

0. 0052200C 

-0.00065700 

•0.01060000 

-0.00110700 

-0.0 1 5I09C8 

-0.00247500 

0,02340000 

-0.00410000 

0.03430600 

-0.00547500 

-0.04240000 

-0.00804600 

-0. 05000000 

-0.01026400 

-0.05695200 

-0. 0 1 260  ^00 

-0.06345910 

-0.01503000 

-0.06452400 

-0.01751300 

-0.07520600 

-0.02003400 

-0.88052100 

-0.02257700 

-8.18552600 

-0.02512000 

-0.09023000 

-0.02767500 

-0.04466500 

-0.03020800 

-0.09865300 

-0.03272000 

-0.10211000 

-0.03520600 

~Q. 10657000 

-0.03765900 

-O.tlOl 3000 

-0.04007600 

-0.11351000 

-0.04245300 

-0.11673000 

-0.04474000 

-0.1 1*80000 

-0.04708400 

-0.12272000 

-0.04433400 

-0.12531000 

-0.05153600 

-0. f 28 ■ IOCG 

-0.0536*600 

-0. i 307)000 

-0.05561 200 

-0. 1 331 7000 

-0. 05786100 

0 . 1 355 1 000 

-0.05440500 

-0. 13?76000 

-0.06 1 88400 

-0.13492000 

-0.06382000 

0  .  14  1  94000 

-0.06571300 

-0.14394000 

-0.0A75o3U0 

-0.14541000 

-0.06437100 

-0.1 4775000 

-0.07113900 

-C. 1 4954806 

-0.07286700 

-0.  I512eu00 

Table  C. 1  Lambda  Parameters 
( continued) 


* 


Vv> 


ALPHA4 

6.2 
*.4 
4.6 
A. 8 
2.0 

2.2 

7.4 

7.4 

7.8 

8.0 

1.4 

1.1 

8. 

9.0 
9. 

9.2 

9.4 
9.6 

9.8 

10.0 

10.2 

10.4 

10.6 
>0.8 
11.0 

11.2 

11.4 

11.6 
11.8 
12.0 

12.2 

12.4 

12.6 
12.8 
13.0 

13.2 

13.4 

13.4 

13.8 
14. 

14.2 

14.4 

14.4 
14.0 
13.0 

13.2 

15.4  -0.415 

15.6  -0.409 

15.8  -0.403 


ALPHA! 
UMil>A7 
0.23130000 
0.29980000 
0.34300000 
0.33950000 
0.33930000 
0.33490000 
0.34830000 
0.34100000 
0. 3 J330000 
0.32550000 
0.00839700 
0.00414700 
0.00204100 
-0.00108100 
-0.00547300 
-0.01130000 
-0.02230000 
-0.03300000 
-0.04350000 
-0.03370000 
-0.04370000 
-0.07340000 
-0.08280000 
-Q. 09200000 
-0.10100000 
-0.10970000 
-0.11820000 
-0. 126400U0 
-0.13450000 
-0.14230000 
-0. 14990000 
-0.13730000 
-0.16430000 
-0,17130000 
-0.17840000 
-0. 18500000 
-0.19150000 
-0.19790000 
-0.20400000 
-0.21000000 
-Q. 21590000 
-0. 22140000 
-0.22720000 
-0 .23260000 
-0.23600000 
-0.24320000 
-0.24810000 
-0.25120000 
-0.25600000 


*  2.00 

LAM  9  A  3 
6.50470000 
4.90190000 
8.31150000 
10.34000000 
12. 39BOOOOO 
14.50800000 
16.731 00000 
19.11700000 
21.71200000 
24.56200000 
0.0 
0.0 

0.00000001 
-0.00000407 
-0.00007075 
-0.OOO2720Q 
0.00101200 
-0.00212560 
-0.00353700 
-0.00518700 
-0. 00702  700 
-0.00901600 
-0.01112200 
-0.01331600 
-0.01337400 
-0. 01 788200 
-0.02021708 
-0.02237000 
-0.02492700 
-0.02728100 
-0.02942300 
-0.03194800 
-0.03425000 
-0.03452400 
-0.03876900 
-0.04098000 
-0.04315106 
-0.04529600 
-0.04 ’39600 
-0.04946200 
-0.05148700 
-0.05347200 
■0.03541600 
-0. 05732400 
-0.05920000 
-0.06100000 
*0.06280000 
-0.06460000 
-0.04430000 


L4MOA4 

0.02153700 
0.01917200 
0.19614006 
0.28724000 
1.34295000 
0.37995000 
0.40523000 
0.42235000 
0.43341000 
6.43177000 
0.80654100 
0.00411200 
0.00207000 
-0.00107100 
* 0  .00554700 
-0.01064500 
0.02074000 
-0. 82*77400 
-0.03601200 
-0.04373400 
-0.05267000 
-0.05948700 
-0.04544000 
-0.07143300 
-0.07484400 
-0.01192700 
*0.01471200 
-0.09122400 
-0.09548400 
*0.09931900 
-0.1IJ34Q0I 
-8.11697000 
-1.11041000 
-0.11369000 
-0.11182000 
-0.11910000 
•0.17244000 
-0.12536000 
-0. 12797000 
-1.13044000 
-0.13785000 
-0. 13514000 
0.13735000 
-0.13946008 
-0.14151000 
-0.14J50000 
-0.14530000 
-0.14720000 
•0.14890000 


LAM0A1 

3.363 
2.638 
2.106 

1.915 
1  .663 
1.661 
I  .881 

1.916 
1.961 
2.014 

-1.009 
-1.004 
9  -1.002 

-Q. 993 
1  -Q. 974 

-0.949 
-0.905 
-0.863 
-0.828 
-0.796 
-Q.766 
-0.738 
-0.713 
-0.689 
-0.668 
-0-648 
-0.629 
-0.611 
-Q.593 
-0.580 
-0.565 
-0.552 
-0.539 
-0.527 
-0.515 
-0.505 
-Q.494 
-0.483 
-0.475 
0  -0.467 

-0.458 
-0.450 
-0.443 
-0.433 
-0.428 
-0.422 
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ABSTRACT 


This  paper  discusses  a  multidisciplinary  study 
conducted  to  select  a  site  for  a  nuclear  power 
plant.  A  series  of  screening  steps  were  car¬ 
ried  out  to  identify  candidate  sites  for  the 
plant,  as  well  as  candidate  water  sources. 
Multiobjective  decision  analysis  methods  were 
used  to  evaluate  and  rank  these  candidate  sites 
and  water  sources;  a  risk  analysis  was  carried 
out  to  determine  the  effects  of  major  uncer¬ 
tainties.  The  evaluation  concerns  considered 
in  the  study  are  discussed,  as  well  as  the 
scales  used  to  measure  impacts  with  regard  to 
these  concerns. 


INTRODUCTION 


An  investor-owned  electric  power  company  in  a 
semi-arid  region  of  the  United  States,  here¬ 
after  referred  to  as  the  Electric  Company, 
planned  to  select  a  site  for  a  proposed  nu¬ 
clear-powered  generating  plant  to  begin  commer¬ 
cial  operation  in  the  late  1980 's  with  a  nomin¬ 
al  capacity  of  2500  megawatts.  Although  the 
Electric  Company  had  not  made  a  definite  deci¬ 
sion  to  build  a  nuclear  power  plant,  it  wished 
to  determine  a  site  so  that  the  nuclear  gener¬ 
ating  option  could  be  compared  with  other  pos¬ 
sible  strategies  for  meeting  the  Electric  Com¬ 
pany's  future  need  for  electric  power. 

Woodward-Clyde  Consultants  was  retained  to  de¬ 
velop  and  implement  a  methodology  to  recommend 
a  site  for  this  plant.  The  problem  had  a  num¬ 
ber  of  challenging  features,  including: 

1)  Difficult  geology:  much  of  the  region  in 
or  near  the  Electric  Company’s  service 
area  is  seismically  active  or  has  other 
natural  features  that  might  make  it  unac¬ 
ceptable  for  a  nuclear  power  plant. 

2)  Limited  water:  any  power  generating  fa¬ 
cility  of  this  size  requires  large  quan¬ 
tities  of  water  for  cooling  purposes,  and 
water  is  in  short  supply  in  the  area. 

3)  Significant  uncertainties:  these  include 
uncertainties  about  geology,  water  avail¬ 
ability,  and  future  socioeconomic  develop¬ 
ments  in  the  area. 

4)  Multiple  siting  concerns:  in  addition  to 
system  costs,  other  significant  siting 


concerns  include  licensing  requirements, 
public  health  and  safety,  environmental 
and  socioeconomic  effects,  and  public 
acceptance. 

5)  Multiple  interest  groups:  the  Electric 
Company  has  responsibilities  to  both  its 
shareholders  and  rate  payers;  in  addition, 
a  variety  of  other  groups  are  interested 
in  nuclear  power. 

6)  Data  limitations:  although  the  Electric 
Company  was  willing  to  put  substantial 
resources  into  the  site  selection,  there 
were  many  data  that  could  not  be  collected 
within  a  realistic  budget  and  schedule  or 
that  were  not  available. 

7)  Regulator^  requirements:  regulations  of 
the  U.S.  Nuclear  Regulatory  Commission  and 
other  government  bodies  impose  various  re¬ 
quirements  on  the  process  that  is  used  to 
select  sites. 

Previous  site  selection  studies,  discussed  be¬ 
low,  had  utilized  multiobjective  decision  anal¬ 
ysis  [6,9]  within  a  structured  system  analysis. 
Although  these  earlier  selection  problems  did 
not  have  all  the  complexities  of  the  present 
one,  decision  analysis  could,  in  concept,  ad¬ 
dress  all  the  issues  listed  above. 


PREVIOUS  RELATED  WORK 


Keeney  and  Nair  [4,5]  and  Keeney  [2,3]  have  ad¬ 
dressed  site  selection  using  decision  analysis. 
Conceptually,  this  approach  proceeds  as  fol¬ 
lows:  First,  objectives  are  established  and 

evaluation  measures,  or  attributes,  X  ,X . 

X^  are  determined  to  measure  the  desirability 
or  any  site  with  regard  to  each  of  the  multiple 
siting  concerns.  A  multiattribute  utility 
function  u(x  ,x2 , . . . ,xn) ,  where  x  is  a  specif¬ 
ic  level  of  X. ,  is  assessed  to  encode  in  mathe¬ 
matical  form  the  company’s  attitude  toward  risk 
taking  and  tradeoffs  among  the  X  .  Next, 
screening  criteria  are  established  and  applied 
to  eliminate  all  areas  that  are  unacceptable 
(for  example,  because  they  are  national  parks 
or  contain  active  faults). 

For  the  area  that  remains  after  screening, 
probability  distributions  p(x  ,x2,...,x  >a), 
where  a  is  a  specific  geographic  location,  are 
assessed  to  encode  in  mathematical  form  the 
available  knowledge  about  the  desirability  of 
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each  location  with  respect  to  all  the  evalua¬ 
tion  concerns.  Where  necessary,  professional 
judgment  is  used  to  assess  the  probability  dis¬ 
tributions  . 

Finally,  based  on  the  axioms  of  decision  anal¬ 
ysis  [8],  the  possible  sites  are  ranked  accord¬ 
ing  to  their  expected  utilities: 

n  (x^  >*2*  •  •  * » ^^)  P  >^2*****  j  a) , 
X1,x2,...,xn 

and  sensitivity  analysis  is  carried  out  to  de¬ 
termine  how  variations  in  u  and  p  affect  the 
relative  ranking  of  sites. 

Conceptually,  this  approach  handles  all  the 
complexities  discussed  in  the  last  section;  as 
a  practical  matter,  the  area  that  remains  after 
screening  generally  is  too  large  to  be  com¬ 
pletely  analyzed.  It  is  not  feasible  to  col¬ 
lect  the  necessary  data  for  a  defensible  as¬ 
sessment  of  the  probability  distributions  p  for 
the  entire  region.  Furthermore,  computer  soft¬ 
ware  is  not  available  to  carry  out  the  expected 
utility  calculations  and  required  sensitivity 
analysis  for  this  many  sites. 

It  is  therefore  necessary  to  screen  out  consid¬ 
erably  more  area  than  what  is  clearly  unaccept¬ 
able  to  reduce  the  remaining  area  to  an  analyz- 
able  size.  For  example,  a  screening  criterion 
might  be  used  that  no  site  could  be  within  one 
mile  of  a  paved  road — the  basic  concern  being 
that  under  certain  conditions  a  fog  resulting 
from  cooling  tower  operation  might  reduce  visi¬ 
bility  on  the  road  to  dangerous  levels.  Note, 
however,  that  this  potential  difficulty  could 
be  reduced  by  proper  design  of  the  plant,  al¬ 
though  this  might  add  to  the  cost  of  the  facil¬ 
ity.  Certain  locations  within  a  mile  of  a 
paved  road  might  have  other  desirable  charac¬ 
teristics  (e.g.,  good  geology  for  construction) 
that  would  make  it  beneficial  to  locate  a  plant 
there,  even  if  additional  money  has  to  be  spent 
to  reduce  fogging.  By  screening  based  on  a 
one-mile  distance  from  roads,  this  type  of  spe¬ 
cial  situation  could  not  be  considered. 

To  make  the  analysis  practical,  in  applications 
screening  criteria  are  introduced  that  elimin¬ 
ate  areas  with  one  or  more  undesirab le  charac¬ 
teristics,  even  if  these  do  not  make  a  site 
completely  unacceptable.  This  reduces  the  area 
to  be  considered,  but  there  is  generally  still 
too  much  area  to  be  analyzed  in  a  practical 
situation.  Thus,  as  a  final  step  before  carry¬ 
ing  out  the  complete  decision  analysis,  profes¬ 
sionals  familiar  with  the  engineering  and  envi¬ 
ronmental  requirements  for  a  plant  site  select 
a  relatively  small  number  of  specific  locations 
(usually  ten  to  twenty)  within  the  area  remain¬ 
ing  after  screening  for  consideration  in  the 
complete  decision  analysis. 

Underlying  this  approach  to  site  selection  is 


the  implicit  assumption  that  enough  potentially 
acceptable  sites  for  the  plant  exist  in  the 
region  of  interest  so  that  screening  for  unde¬ 
sirable,  as  well  as  for  unacceptable,  charac¬ 
teristics  will  not  seriously  limit  the  useful¬ 
ness  of  the  results .  Recent  developments  in 
computerized  geographic  information  processing 
give  promise  of  reducing  the  need  for  this  type 
of  analysis  "shortcut".  These  developments 
will  be  discussed  in  more  detail  below. 


ANALYSIS  PROCESS  FOR  THIS  STUDY 


The  general  approach  outlined  in  the  last  sec¬ 
tion  was  applied  in  the  study  discussed  below 
to  reach  a  recommendation  for  a  suitable  nucle¬ 
ar  power  plant  site.  However,  early  in  the 
analysis  process  we  realized  that  finding  an 
acceptable  site  which  also  had  a  source  of 
water  sufficient  to  supply  the  cooling  needs  of 
the  plant  would  be  unlikely.  For  this  reason, 
it  was  decided  to  decouple  the  initial  search 
for  a  site  from  the  search  for  a  source  of 
water.  Once  reasonable  candidate  sites  and 
candidate  water  sources  had  been  located,  these 
would  be  coupled  and  evaluated  as  units. 

In  another  modification  of  the  site  selection 
process  outlined  in  the  last  section,  we  dis¬ 
covered  when  we  ranked  our  candidate  sites  and 
carried  out  a  sensitivity  analysis  that  certain 
risks  needed  to  be  evaluated  further  to  reach  a 
final  conclusion  about  the  most  preferable 
site.  Thus,  an  extra  step  of  risk  analysis  was 
added  after  the  initial  ranking. 

The  final  site  selection  process  that  was  used 
is  diagrammed  in  Figure  1.  Each  of  the  steps 
shown  in  this  figure  will  now  be  discussed. 

Determine  Evaluation  Concerns.  As  noted  ear¬ 
lier,  the  Electric  Company  had  not  made  a  def¬ 
inite  decision  to  build  a  nuclear  power  plant; 
however,  it  wished  to  keep  this  option  open  as 
long  as  it  appeared  viable.  The  primary  intent 
of  our  study  was  to  obtain  early  assurance  that 
a  potentially  licensable  nuclear  power  plant 
site  would  be  available  prior  to  the  Electric 
Company  making  a  major  financial  commitment  to 
nuclear  power.  Thus,  the  principal  objectives 
of  the  siting  study  were  as  follows: 

1)  Identify  candidate  sites  for  a  nuclear 
power  plant  that  could  be  licensed  and 
constructed  in  approximately  ten  years, 

2)  Identify  candidate  sites  that  meet  the 
rules  and  regulations  of  the  U.S.  Nuclear 
Regulatory  Commission  [11,12,13], 

3)  Identify  candidate  sites  that  are  among 
the  best  that  could  reasonably  be  found  in 
the  region  of  interest,  and  • 

4)  Select  from  among  the  candidate  sites  a 
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Figure  1.  SITE  SELECTION  PROCESS 


primary  site  and  at  least  one  alternative 
site  such  that  no  other  candidate  sites 
are  obviously  superior  to  them. 

In  order  to  meet  these  objectives,  three  over¬ 
all  areas  of  concern  were  identified  as  impor¬ 
tant  : 

1)  Public  Health  and  Safety, 

2)  Environmental  Impact,  and 

3)  System  Cost  and  Reliability. 

The  specific  considerations  that  needed  to  be 
addressed  under  each  of  these  areas  of  concern 
are  discussed  below. 

Determine  Region  of  Interest.  The  geographic 
area  in  which  the  search  for  power  plant  sites 
was  conducted  is  called  the  region  of  interest. 
For  this  study  the  Electric  Company  selected 
its  home  state  as  the  region  of  interest.  This 
state  is  one  of  the  larger  western  U.S.  states, 
and  the  Electric  Company  serves  a  significant 
portion  of  the  state,  including  its  largest 
city.  The  Electric  Company  is  an  investor- 
owned  public  utility  incorporated  in  the  state 
and  is  regulated  by  the  state  Public  Service 
Commission.  The  Electric  Company's  chartered 
responsibilities  are  defined  in  terms  of  the 
state  and  its  inhabitants.  A  region  of  inter¬ 
est  smaller  than  the  state  would  have  placed 
arbitrary  emphasis  on  one  area  over  another. 

A  region  of  interest  larger  than  the  Electric 
Company's  home  state  could  make  it  necessary 
for  the  company  to  deal  with  difficult  legal 
and  practical  questions  with  which  it  has 
little  experience. 

Screen  to  Determine  Candidate  Water  Sources  . 

The  candidate  water  sources  were  selected  by  a 

three-step  screening  process.  The  scope  of 
water  resources  considered  included  both  sur¬ 
face-water  and  ground-water  sources  as  well  as 
other  potential  sources  such  as  reclaimed 
wastewater  from  sewage  treatment  plants  and 
mine  dewatering  operations.  The  primary  objec¬ 
tive  of  the  screening  process  was  to  identify 
candidate  water  sources  for  which  there  was  a 
high  probability  that  acquisition,  development 
and  use  for  power  plant  cooling  would  be  fea¬ 
sible.  A  screening  process  with  three  sequen¬ 
tial  screening  steps  was  used  so  that  as  the 
process  proceeded  sources  with  a  lower  likeli¬ 
hood  of  meeting  the  study  objectives  were  elim¬ 
inated  from  consideration  and  the  remaining 
sources  could  be  examined  in  greater  detail 
within  the  time  and  resource  constraints  of  the 
s  tudy . 

The  screening  criteria  used  during  this  process 
were  as  follows: 

1)  Minimum  Water  Requirements 


* 
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a)  Source  must  be  able  to  supply  at  least 
10,000  acre-feet  per  year  for  40  years. 


2)  Legal  Cons iderat ions 

a)  Source  must  have  unappropriated,  pur¬ 
chasable  or  leasable  water  rights, 

b)  State  Engineer's  administrative  crite¬ 
ria  must  be  straightforward, 

c)  Water  rights  must  be  geographically 
concentrated,  and 

d)  Source  must  have  low  potential  for 
competing  or  conflicting  uses. 

3)  Hydrologic  Data 

a)  Hydrologic  information  must  be  suffi¬ 
cient  to  evaluate  both  the  availability 
of  water  and  its  production  character¬ 
istics  , 

b)  Aquifers  must  have  a  minimum  saturated 
thickness  of  50  feet  (water  table 
aquifers  only),  and 

c)  Aquifers  must  have  a  minimum  average 


well  yield  of  200  gallons  per  minute 
(ground-water  sources  only) . 

These  were  identified  by  hydrologic  specialists 
familiar  with  the  overall  objectives  of  the 
siting  study  as  well  as  the  water  resources  of 
the  region  of  interest.  As  a  result  of  apply¬ 
ing  these  criteria,  six  candidate  water  sources 
were  identified,  including  three  underground 
basins,  two  surface  sources,  and  one  mine  de¬ 
watering  operation. 

Screen  to  Determine  Candidate  Siting  Areas. 
Candidate  areas  for  siting  the  power  plant  were 
selected  by  a  sequential  three-step  screening 
process  similar  in  concept  to  that  used  to  se¬ 
lect  candidate  water  sources.  The  objective  of 
this  screening  process  was  to  select  from  with¬ 
in  the  region  of  interest  those  areas  that  have 
the  highest  likelihood  of  containing  candidate 
sites  that  meet  the  study  objectives  discussed 
earlier. 

Examples  of  the  screening  criteria  used  during 
this  process  are  listed  in  Table  1.  These  cri¬ 
teria  were  identified  by  specialists  familiar 


TABLE  1 

EXAMPLE  SCREENING  CRITERIA  FOR  SELECTION  OF  CANDIDATE  SITING  AREAS 


Issue 

Consideration 

Measure 

Criteria  for  Inclusion 

HEALTH  AND 
SAFETY 

Radiological  release  due 
to  surface  fault  rupture 
at  site 

Distance  to  faults 

Areas  >  5  miles  from  faults 

Collapse  or  subsidence  of 
plant  foundations  due  to 
solution 

Location  of  karst  topo¬ 
graphy  or  soluble 
lithologies 

Areas  where  no  known  karst  or 
solution  potential  exists 

Radiological  release  due 
to  accident  at  (or  in 
connection  with)  nearby 
hazardous  facilities 

Distance  from  airports 

Areas  >  5  miles  from  major 
airports 

ENVIRON¬ 

MENTAL 

IMPACT 

Conflict  with  existing 
or  proposed  land  use 

Location  with  respect 
to  designated  land 
use  areas 

Areas  outside  of  designated 
land  use  areas  larger  than 

1,000  acres 

Potential  for  local 
socioeconomic  impact 

Location  with  respect 
to  population  centers 
and/or  industrial 
development 

Proximity  to  labor  and 
materials  markets 

Loss  or  alteration  of 
important  ecological 
considerations 

Location  with  respect 
to  areas  that  are 
biologically  unique 
or  diverse 

Areas  outside  of  important 
ecological  systems 

SYSTEM 

COST  AND 
RELIABILITY 

Rugged  terrain 

Slope 

Areas  with  less  than  10  percent 
slope 
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with  the  overall  objectives  of  the  siting  study 
as  well  as  the  specific  disciplines  relevant 
for  this  screening.  To  reduce  the  licensing 
uncertainties  and  their  related  financial 
risks,  the  Electric  Company  adopted  screening 
criteria  to  exclude  areas  that  might  present 
licensing  difficulties.  In  particular,  it  was 
possible  that  site-specific  geologic  and  seis- 
mologic  investigations  might  lead  to  unaccept¬ 
able  delays  in  the  licensing  and  construction 
process.  Thus,  geologic  and  seismic  screening 
criteria  were  imposed  such  that  for  areas  not 
screened  out  there  was  a  relatively  high  degree 
of  confidence  that  detailed  site-specific  stud¬ 
ies  could  be  satisfactorily  completed  and  yield 
a  favorable  assessment  within  approximately  one 
year.  Note,  however,  that  areas  excluded  might 
contain  suitable  sites. 

Determine  Candidate  Sites.  As  noted  earlier, 
the  screening  process  did  not  lead  to  specific 
candidate  sites,  but  to  areas  that  have  a  high 
likelihood  of  containing  acceptable  sites. 

Within  these  areas,  specific  candidate  sites 
were  identified  by  qualitative  methods  consid¬ 
ering  site-specific  features.  These  included 
site-specific  considerations  of  cultural  re¬ 
sources,  population  census,  surface  or  subsur¬ 
face  mineral  resources,  land  use  status,  land 
use  patterns,  meteorology,  transmission  and 
pipeline  corridor  feasibility,  site  engineering, 
geological  and  geotechnical  conditions,  nearby 
hazardous  facilities,  socioeconomic  conditions, 
and  terrestrial  and  aquatic  biology. 

To  obtain  necessary  information  to  make  a  site- 
specific  evaluation,  aerial  environmental  and 
engineering  reconnaissance  was  conducted  by  an 
interdisciplinary  team  of  experts.  Based  on 
this  evaluation,  seven  specific  candidate  sites 
were  identified  for  further  study. 

Identify  Candidate  Site  and  Candidate  Water 

Source  Combinations.  As  expected,  the  candi¬ 

date  water  sources  and  candidate  sites  were  not 
at  the  same  locations.  Thus,  to  assess  the 
relative  desirabilities  of  the  various  candi¬ 
date  sites,  it  was  necessary  to  determine  which 
water  source  or  sources  should  be  used  with 
each  site.  This  required  assessing  the  cost  of 
transferring  water  from  the. various  sources  to 
each  site,  as  well  as  the  associated  environ¬ 
mental  effects.  To  do  this  it  was  necessary  to 
lay  out  feasible  pipeline  routes  between  the 
sources  and  sites.  If  only  one  source  is  need¬ 
ed  to  supply  a  site,  there  would  be  a  total  of 
6  sources  x  7  sites  *  42  pipeline  routes  to  be 
laid  out.  In  fact,  some  sources  did  not  have 
sufficient  water  individually  to  supply  all  the 
needs  of  the  proposed  plant,  so  combinations  of 
sources  had  to  be  considered.  The  process  of 
laying  out  pipeline  routes  was  relatively 
straightforward,  but  time  consuming,  and  sixty 
different  source/site  combinations  were  iden¬ 
tified. 

Rank  Site/Source  Combinations.  The  evaluation 


and  ranking  of  the  various  combinations  of  can¬ 
didate  water  sources  and  candidate  sites  was 
done  utilizing  standard  multiobjective  decision 
analysis  methods  [6].  The  use  of  these  to  rank 
sites  has  been  discussed  in  detail  elsewhere 
[2, 3, 4, 5],  so  the  process  will  only  be  outlined 
here.  It  consists  of  the  following  steps: 

1)  Determine  evaluation  measures, 

2)  Assess  a  utility  function  over  these 
measures , 

3)  Determine  the  levels  of  the  various  eval¬ 
uation  measures  that  would  result  from 
selecting  each  site  (uncertainties  about 
these  levels  are  encoded  into  probability 
distributions) , 

4)  Rank  the  sites  in  order  of  their  expected 
utilities,  and 

5)  Carry  out  sensitivity  analysis. 

Note  that  this  approach  is  based  on  well-estab¬ 
lished  principles  of  decision  making  under  un¬ 
certainty  [8]  . 

The  following  specific  factors  were  identified 
by  the  study  staff  ss  important  for  evaluating 
source/site  alternatives: 

1)  System  cost 

2)  Population  near  the  site 

3)  Environmental  impact  of  water  and  electric 
transmission  corridors 

4)  Environmental  impact  of  plant  construction, 
including 

—  socioeconomic  impact 

—  biological  impact 

5)  Environmental  impact  at  the  water  source, 
including 

long-term  socioeconomic  effects 

—  short-term  socioeconomic  effects 

—  biological  impact 

—  loss  of  irrigable  land. 

The  evaluation  measures  developed  for  these  fac¬ 
tors  are  shown  in  Table  2. 

It  waa  established  through  standard  preferential 
and  utility  independence  checks  [6]  that  an  ad¬ 
ditive  utility  function 

9 

u<xrx2 . V  "  2  vw 

i-1 

was  appropriate,  where  the  x  are  evaluation 
measures,  the  \i±  are  single  attribute  utility 
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TABLE  2 

EVALUATION  MEASURES  FOR  SITE  RANKING 


Evaluation 

Measure_ Definition 


X^:  Levelized  system  cost  Levelized  annual  cost  in  year  0  dollars.  This  includes 

water-sensitive  costs,  site-sensitive  costs,  and  baseline 
reference  costs. 


X^:  Site  population  Site  population  factor  [4,5,7]. 

X^:  Biological  impact  at  A  13-point  constructed  scale,  as  illustrated  in  Table  3a. 
plant  site 


X^:  Biological  impact  at  A  3-point  constructed  scale,  as  illustrated  in  Table  3b. 
water  source 


X^:  Environmental  impact 

of  corridors 


A  weighted  sum  of  the  distances  for  the  electrical  trans¬ 
mission  and  water  supply  pipeline  corridors,  using  the 
weighting  factors  illustrated  in  Table  4. 


X^:  Socioeconomic  effects 
at  plant  site 


0  Annual  population  growth  rate  due  to  peak-year  con¬ 
struction  activities  is  less  than  15  percent.  This  in¬ 
dicates  that  there  are  population  centers  near  the  site 
with  existing  infrastructure  to  serve  as  a  base  for  the 
new  population  influx. 


1  Annual  population  growth  rate  due  to  peak-year  con¬ 
struction  activities  is  more  than  15  percent.  This  in¬ 
dicates  that  there  are  no  existing  population  centers 
of  significance;  boom-bust  development  is  virtually 
certain  to  occur. 


X^:  Short-term  socioeco-  The  value  of  production  directly  affected  by  the  withdrawal 

nomic  effects  at  water  of  irrigation  due  to  diversion  of  water  to  the  power  plant, 
source  in  year  0  dollars. 


xg:  Long-term  socioeco-  For  each  water  source,  the  scale  used  is:  (aggregate  annual 

nomic  effects  at  water  personal  income  per  annual  quantity  of  water  consumption) 

source  X  (quantity  of  water  supplied  to  the  power  plant  from  the 

water  source)  in  levelized  year  0  dollars. 


Xg:  Loss  of  potentially  Weighted  square  miles  of  cropland  that  could  potentially  be 

irrigable  land  retired  from  use  at  a  candidate  water  source,  using  the 

following  weighting  factors  on  area: 

1  Cropland  currently  under  irrigation 

1  Land  highly  suitable  for  irrigation 

0.7  Land  moderately  suitable  for  irrigation 
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TABLE  3 

EXAMPLE  DEFINITIONS  OF  LEVELS  FOR  RANKING  EVALUATION  MEASURES  X  AND  X 

3  4 


a)  Example  Definitions  of  Levels  of  Biological  Impact  at  Plant  Site  (X3) 

Level  Definition  

Removal  of  6  square  miles  having  >  25  percent  of  cultivated  agricultural  use. 

4  Removal  of  6  square  miles  of  grassland,  shrubland,  or  piny on- juniper  habi¬ 

tat  that  includes  <_  10  percent  riparian  or  wetland  habitat. 


8 


Removal  of  6  square  miles  of 
within  1  mile  of  significant 
which  £  25  percent  is  actual 
or  otherwise  unique  species. 


grassland,  shrubland,  or  pinyon-j uniper  habitat 
actual  or  potential  raptor  habitat,  and  of 
or  potential  habitat  for  threatened,  endangered, 


Removal  of  6  square  miles  of  grassland,  shrubland,  or  pinyon-j uniper  habitat 
within  1  mile  of  significant  actual  or  potential  raptor  habitat  and  including 
1  10  percent  riparian  or  wetland  habitat  and  >  25  percent  actual  or  potential 
habitat  for  threatened,  endangered  or  otherwise  unique  species. 


b)  Example  Definitions  of  Levels  of  Biological  Impact  at  Water  Source  (X^) 

Definition 


Lowering  of  water  table  beneath  cultivated  agriculture;  the  basin  supports 
only  non-phreatophytic  vegetation  and  no  riparian  or  wetland  habitat;  or 
1  ^  percent  of  the  basin  supports  phreatophytic  vegetation  or  riparian  or 
wetlands  habitats,  but  these  habitats  do  not  support  threatened,  endangered, 
or  otherwise  unique  species;  essentially  no  impacts. 


Lowering  of  water  table  beneath  a  basin  with  >  5  percent  of  the  area  support¬ 
ing  phreatophytic  vegetation  or  riparian  or  wetland  habitat;  the  phreatophytic 
vegetation  or  riparian  or  wetland  habitat  supports  or  potentially  supports 
threatened,  endangered,  or  otherwise  unique  species;  or  the  basin  contains 
springs  which  support  threatened,  endangered,  or  otherwise  unique  species. 
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TABLE  4 

EXAMPLE  WEIGHTING  FACTORS  FOR  RANKING  EVALUATION  MEASURE 


a)  Example  Weighting  Factors  for  Electrical  Transmission  Lines 


* 

Raw  Mileage 

Weighted  Mileage 

Criteria 

«r 

1 

1 

Route  traversing  unpopulated  rangeland;  not  visible 
from  highways  or  high-use  roadway.  Route  not  affect¬ 
ing  any  known  endangered  species  or  important  limited 
habitats;  does  not  intrude  on  a  "pristine",  historic, 
or  culturally  significant  area. 

1 

2 

Route  traversing  populated  areas.  Route  traversing 
Bureau  of  Land  Management  or  public  lands,  not  util¬ 
izing  an  existing  corridor.  Route  having  aesthetic 
intrusion  on  primary  highways  and  high-use  roadways 
(parallel  to  and/or  visible  from  highway);  or  aesthetic 
intrusion  on  a  national  or  state  monument  or  park. 

1 

10 

Route  traversing  state  or  national  parks  or  monuments, 
military  bases,  or  military  research  areas.  Route 
traversing  ecologically  sensitive  wetlands  and  migra¬ 
tory  wildfowl  refuges;  or  habitats  containing  unusual 
or  unique  communities,  endangered  species,  or  intro¬ 
duced  game  species . 

b)  Example  Weighting  Factors  for  Water  Supply  Pipelines 


Raw  Mileage  Weighted  Mileage 


1 

1 

Route  traversing  unpopulated  rangeland;  utilizing  exis- 
tin8  Industrial  corridor.  Route  not  affecting  any 
known  endangered  species  or  important  limited  habitats. 
Route  does  not  intrude  on  a  "pristine",  historic, 
culturally  significant,  or  archaeologic  and  paleonto- 
logic  resource  area. 

1 

3 

Route  traversing  state  or  federal  forested  lands,  wild¬ 
life  management  or  critical  habitat  areas.  Route  tra¬ 
versing  ecologically  sensitive  wetlands. 

1 

10 

Route  traversing  state  or  national  parks  or  monuments, 
military  bases,  or  military  research  areas.  Route  tra¬ 
versing  habitats  containing  unusual  or  unique  communities, 
endangered  species,  or  introduced  game  species. 
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functions,  and  the  are  scaling  constants. 

The  and  k^  were  assessed  using  standard  pro¬ 
cedures  [6]  as  follows: 

1)  The  lead  specialist  for  a  specific  evalua¬ 
tion  measure  assessed  the  u  over  that 
measure 

2)  The  multidisciplinary  group  of  specialists 
meeting  together  assessed  the  k^ 

3)  The  resulting  utility  function  was  re¬ 
viewed  by  management  of  the  Electric 
Company . 

As  would  be  expected  when  persons  representing 
such  diverse  specialties  are  involved,  the  as¬ 
sessment  of  the  k^  proved  to  be  complex.  A 
meeting  lasting  more  than  eight  hours  was  need¬ 
ed  to  reach  general  agreement  (but  not  consen¬ 
sus)  on  the  nine  scaling  constants.  A  major 
disagreement  concerned  the  amount  of  weight 
that  should  be  attached  to  measure  Xg  (long¬ 
term  socioeconomic  effects  at  water  source) . 

Some  argued  that  a  dollar  of  Xft  should  be 
counted  equally  with  a  dollar  of  X  (levelized 
system  cost),  while  others  argued  that  Xft 
should  have  a  weight  of  zero.  It  was  decided 
to  settle  this  disagreement  by  carrying  out  a 
sensitivity  analysis  that  covered  this  wide 
range  of  possible  weights  for  Xg. 

An  extensive  data  collection  effort  was  carried 
out  to  estimate  the  levels  of  the  measures 
Xl,X2,...,X9  associated  with  each  water  source/ 
site  combination.  This  included  local  data 
collection  in  the  areas  of  the  water  sources 
and  sites.  In  addition,  a  detailed  cost  anal¬ 
ysis  for  the  power  plant,  electric  transmission 
line,  water  supply  pipeline  and  well  field  was 
carried  out  as  a  joint  effort  by  the  Electric 
Company,  Woodward-Clyde  Consultants  and  the 
Bechtel  Power  Corporation.  This  was  used  to 
determine  levels  for  X^  (levelized  system  cost)  • 

In  cases  where  there  was  significant  uncertain¬ 
ty,  standard  probability  assessment  procedures 
[10]  were  used  to  encode  this  uncertainty  using 
the  judgments  of  relevant  experts. 

The  utility  function  and  assessed  probability 
distributions  for  the  evaluation  measures  were 
analyzed  using  a  standard  multiobjective  deci¬ 
sion  analysis  computer  program  [14] .  Extensive 
sensitivity  analysis  was  carried  out,  and  it 
was  concluded: 

1)  The  weight  attached  to  Xg  (long-term  soci¬ 
oeconomic  effects  at  water  source)  did  not 
influence  the  ranking  of  the  more  prefer¬ 
able  water  sources  or  sites. 

2)  Two  of  the  seven  candidate  sites  were  more 
preferable  than  the  others  over  a  wide 
range  of  assump tions  about  which  water 
source  would  be  used. 


3)  One  of  the  surface  water  sources  was  more 
preferable  from  a  cost  and  environmental 
standpoint  than  the  other  five  candidate 
water  sources  for  a  wide  range  of  assump¬ 
tions  about  which  site  would  be  utilized; 
however,  for  legal  and  political  reasons 
it  was  uncertain  whether  this  source  could 
be  utilized. 

4)  If  the  most  preferable  water  source  could 
not  be  utilized,  then  a  specific  one  of 
ground-water  sources  was  the  next  most 
preferable  candidate  water  source  for  a 
wide  range  of  assumptions  about  which  site 
would  be  used. 

5)  The  relative  ranking  of  the  top  two  candi¬ 
date  sites  was  most  strongly  influenced 
by: 

—  whether  or  not  the  most  preferable 
(surface)  candidate  water  source 
could  be  utilized, 

—  specifically  how  much  cooling  water 
would  be  needed  for  the  power  plant, 
and 

—  the  relative  likelihood  of  local  sup¬ 
port  or  opposition  toward  a  nuclear 
power  plant  at  each  of  the  two  sites. 

Carry  Out  Risk  Analysis.  In  order  to  analyze 
more  fully  the  influence  of  the  factors  listed 
under  item  5  in  the  last  paragraph,  a  formal 
risk  analysis  was  carried  out.  A  decision  tree 
[1,9]  was  constructed  to  represent  the  choice 
between  the  two  top-ranked  candidates  sites. 
There  was  significant  disagreement  among  the 
relevant  experts  about  the  probabilities  asso¬ 
ciated  with  the  various  factors  listed  under 
item  5  above  for  each  of  the  two  sites.  How¬ 
ever,  the  risk  analysis  showed  that  one  of  the 
two  sites  was  more  preferred  for  all  of  the 
various  probability  levels  that  were  proposed 
by  the  different  experts. 

Identify  Preferred  Site.  Based  on  the  results 
of  the  risk  analysis,  as  well  as  the  earlier 
analysis,  one  site  was  identified  as  most  pre¬ 
ferred.  A  clear  second  most  preferable  choice 
was  al30  identified.  In  addition,  two  sources 
of  water  were  identified  as  most  preferable, 
as  noted  earlier. 


RESULTS  OF  THE  ANALYSIS 


Following  the  analysis  discussed  above,  the 
Electric  Company  took  actions  to  obtain  rights 
for  land  at  both  of  the  top-ranked  sites.  In 
addition,  it  pursued  steps  to  obtain  rights  to 
water  from  the  top  two  water  sources.  In  par¬ 
ticular,  land  and  water  rights  were  obtained 
for  the  ground-water  source  and  a  testing  pro¬ 
gram  was  initiated  to  determine  more  accurately 
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how  much  water  is  available  there. 


Prentice-Hall,  1979. 


* 


With  the  recent  regulatory  and  public  opposi¬ 
tion  to  nuclear  power,  the  Electric  Company  has 
slowed  its  timetable  for  pursuing  the  nuclear 
option,  and  shifted  greater  emphasis  to  fossil- 
fueled  generation  options.  However,  active 
work  continues  on  developing  the  water  sources 
identified  in  this  study  for  possible  use  with 
some  type  of  generation  facility. 


CONCLUDING  COMMENTS 


The  site  selection  study  outlined  above  repre¬ 
sented  a  significant  system  analysis  effort  in¬ 
volving  over  50  professionals  in  several  divi¬ 
sions  of  three  companies  and  was  carried  out 
over  a  period  of  more  than  a  year.  The  system¬ 
atic  analysis  procedures  discussed  above  were 
important  in  completing  the  study  and  reaching 
defensible  and  understandable  conclusions. 

As  noted  earlier,  the  screening  procedures  used 
allowed  the  analysis  to  be  carried  out  in  a 
practical  and  timely  manner.  However,  they 
could  potentially  eliminate  areas  from  consid- 
eration  that  might  be  desirable  from  an  overall 
standpoint.  This  could  occur  if  an  area  failed 
to  pass  one,  perhaps  not  very  significant, 
screen  while  being  very  desirable  with  regard 
to  a  variety  of  other  evaluation  concerns. 

Computerized  geographic  information  systems 
that  are  new  starting  to  come  into  use  will 
allow  consideration  of  this  type  of  tradeoff. 
With  these  systems,  geographically-orientad 
information  of  the  type  considered  in  this 
study  can  be  entered  into  a  computerized  data 
base  and  sophisticated  analysis  can  be  done. 
These  methods  should  help  to  reduce  both  the 
need  for  screening  and  the  amount  of  tedious 
and  error-prone  hand  map  work  now  necessary. 
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APPLICATIONS  PANELS  SUMMARY 


The  session  on  "Practical  Aoolication  of  Risk 
and  Uncertainty  Analysis,"  oresented  real 
world  orojects,  Programs  and  case  studies. 
While  the  published  oapers  reflected  the  oer- 
sonal  experiences  and  research  efforts  of  the 
authors,  it  is  difficult  to  adeouately  des¬ 
cribe  the  dynamic  interaction  and  intellec¬ 
tual  stimulation  that  occurred  during  the 
panel  discussions  and  after  the  presentation 
of  each  naper. 

The  use  of  small  groups,  composed  of  knowl¬ 
edgeable  members  from  industry,  government 
and  academe,  interacting  on  a  defined  sub¬ 
ject  for  a  lengthy  period  of  time  (over  two 
hours  in  most  panels),  led  to  a  much  greater 
depth  of  discussion,  personal  involvement, 
exchange  of  ideas  and  cross  fertil ization 
than  rarely  occurs  in  a  orofessional  meet¬ 
ing.  One  lively  exchange  centered  around 
how  to  incorporate  the  behavioral  aspects 
into  an  apnlication  of  risk  and  uncertainty 
analysis . 

While  papers  ranged  from  programs  and  oro¬ 
jects  in  functional  areas  of  construction 
and  nuclear  plant  site  selection  to  DoD 
contracting  case  studies,  the  oanels  con¬ 
cluded  there  was  considerable  value  in  re¬ 
viewing  the  applications  of  risk  and  uncer¬ 
tainty  analysis . 

Applications  of  risk  and  uncertainty  analy¬ 
sis  techniques  included;  mul tiobjecti ve 
decision  analysis  to  evaluate  alternatives, 
a  risk  analysis  to  determine  the  effects  of 
major  uncertainties,  decision  tree  and  multi  - 
attribute  utility  and  modeling,  and  the  deve¬ 
lopment  of  a  model  to  assist  government  con¬ 
tract  negotiators  (price  analysts)  in  cooing 
with  uncertainty. 

Models  were  described  involving  both  para¬ 
metric  and  random  variable  terms,  and  ap¬ 
plying  the  RS  distribution  to  represent 
unimodal  random  variables.  In  addition, 
comouter  simulation  was  used  to  model  com¬ 
plex  bidding  environments. 

The  applications  presented  were  based  on  real 
world  projects.  The  degree  of  success  varied 
in  the  studies  described.  Application  of  the 
tools  was  discussed  in  terms  of  how  well  they 
achieved  the  study's  major  objectives. 

The  following  comments  are  illustrative  of 
the  degree  to  which  applications  were  con¬ 
sidered  successful : 

1.  The  credibility  of  multiobjective  deci¬ 
sion  analysis  as  a  valuable  tool  has 
increased. 


2.  The  issues  of  organization  of  the  studv 
effort  and  channels  of  communication  with 
the  parties  involved  can  be  explicitly 
treated  and  adjusted  as  an  analysis 
proceeds . 

3.  The  multiobjective  decision  analysis  pro¬ 
vided  a  recognition  that  a  single  best 
answer  may  not  be  a  desirable  outcome. 

The  value  of  the  tool  mav  be  the  indica¬ 
tion  that  a  few  alternatives  are  clearly 
superior  to  many  others. 

4.  The  government  price  analyst  (PA)  in 
contract  negotiations  is  a  key  decision 
point  when  attempting  to  imolement  change 
and  generate  increased  attention  to  risk 
and  uncertainty. 

5.  The  binomial  distribution  can  be  used  to 
transform  long-term  bidding  orobabi li ties 
into  short-term  bidding  orobabilities. 
Also,  it  can  be  used  to  determine  which 
long-term  oricing  nolicies  are  best  in 
the  long  run  and  which  are  best  in  the 
short  run. 

6.  Multiobjective  decision  analysis  tools 
nrovide  a  useful  aid  in  the  management 
of  risk  after  the  initial  purpose  of  the 
analysis  (the  selection  of  the  desired 
alternative)  has  been  achieved. 

The  1 imi tattoos  or  weakness  of  the  applica¬ 
tions  and  techniques  discussed  and  analyzed 

by  oresentators  include  the  following: 

1.  The  range,  point  estimates  and  probabil¬ 
ity  distributions  are  not  required  or 
nrovided  for  in  contractor  oricing  data. 
Such  data  is  not  generated  or  reviewed 

by  the  orocurement  agency,  unless  techni¬ 
cal  exnerts  are  available  in  their  ser¬ 
vice  or  branch. 

2.  Nonlinear  regression  programs,  such  as 
8MDR-3R,  are  unable  to  handle  models 
containing  two  or  more  nonlinear  inde¬ 
pendent  variables. 

3.  Simulation  programs  typically  are  not 
able  to  handle  correlated  distributions. 

4.  Missing  data  in  regression  studies  re¬ 
sults  in  the  inability  to  estimate 
correctly  the  values  of  parameters. 

5.  Specialists  responsible  for  major  risk 
and  uncertainty  analyses  should  not  be 
subordinate  to  the  program  manager. 
Comoetent  study  results  are  frequently 
altered  to  yield  policies  to  meet  the 
self-interests  of  suoeriors. 

6.  The  organizational  placement  of  the  de¬ 
cision  analysis  in  relation  to  the  pro¬ 
ject  director  and  the  client  is  a  crucial 
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issue.  There  is  a  need  for  an  apprecia¬ 
tion  of  the  role  and  the  limits  of  the 
analyst. 

7.  There  is  an  inherent  difficulty  in  repre¬ 
senting  the  "decision-maker"  and  "his" 
utility  function.  This  may  be  treated 
explicitly,  however,  and  can  be  subjected 
to  a  sensitivity  analysis. 

8.  The  process  of  model  building  frequently 
does  not  include  the  decision-maker  in  the 
process. 

9.  Program  management  must  be  responsible 
for  the  application  of  risk  and  uncertain¬ 
ty  analysis  to  a  oroject.  Schedules  and 
budgets  can  be  exceeded  by  substantial 
amounts  unless  risk  and  uncertainty  are 
considered. 

RECOMMENDATIONS 

Each  panel  was  given  the  task  of  recommending 
methods  for  improving  applications  of  risk  and 
uncertainty  analysis  to  the  real  world  of 
large  programs.  The  limitations  and  weak¬ 
nesses  summarized  above  in  themselves  consti¬ 
tute  a  substantive  list  of  the  problems  and 
issues  which  can  be  used  for  further  study  and 
research.  There  were  other  specific  recommenda¬ 
tions  made  by  the  panels.  The  major  ones,  not 
in  any  order  of  importance,  dealing  with 
aoplications  were: 

1.  Exneriment  with  or  model  a  new  program 
during  contract  negotiations.  This  would 
permit  the  government  pricing  analyst  the 
option  of  when  to  structure  FPI  and  CPIF 
contracts  and  the  amounts  for  a)  share 
ratios,  b)  target  cost,  c)  target  profit 
rates ,  and  d)  cei 1 ings . 

2.  The  need  to  generate  or  perform  a  decision 
analysis  impact  statement  to  recognize 

the  scope  and  limitations  of  the  analytical 
methods  to  be  used.  Can  alternatives 
safely  be  screened  out  and  limit  the 
analysis  so  that  a  project  manager  can 
retain  an  intuitive  “feel."  Should  the 
search  for  larger  models  and  processors 
such  as  computerized  geographic  information 
processing  or  system  dynamics  to  accomo¬ 
date  the  more  complex  issues  be  continued? 

3.  There  is  a  need  to  better  identify  and 
catalogue  risk  considerations  to  be  managed 
■fol lowing  the  decision  analysis.  The  pur¬ 
pose  is  to  devise  strategies  and  techni¬ 
ques  to  cope  with  risk  and  uncertainty, 
perhaps  through  approaches  such  as  the 

fol lowing: 

a)  Apply  additional  resources  to  tasks 
with  high  risk. 

b)  Keep  multiple  alternatives  as  an 


open  hedge. 

c)  Aonly  "cost  containment,"  i.e.  "if 
it  fails,  minimize  the  impact  of 
failure." 

4.  Explore,  study  and  research  other  ways  to 
measure  and  assess  risk.  In  particular, 
oossibilistic  risk  assessment  (fuzzy  sets) 
should  be  investigated. 

5.  Study  organizational  behavior  with  respect 
to  the  role  and  positioning  of  the  analyst 
performing  risk  and  uncertainty  studies. 

6.  Develop  both  training  and  graduate  degree 
orograms  to  generate  a  trained  core  of 
specialists  regularly  engaged  in  analysis 
and  direct  thesis  and  dissertation  research 
toward  DoD  risk  and  uncertainty  problems 
and  issues. 

7.  Elicit  DoD  support  for  the  establishment  of 
a  designated  national  data  bank  for  the 
study  of  risk  and  uncertainty. 

8.  Have  the  decision-maker  agree  on  the  defi¬ 
nition  of  the  problem  to  be  solved  early 
in  the  orogram. 

9.  The  decision  strategy  must  reflect  the  fact 
that  systems  analysis  should  take  priority 
over  analysis  of  independent  events. 

10.  Additional  study  and  research  should  be 
conducted  on  both  DoD  and  defense  industry 
program  manager's  decision-making  approach¬ 
es  to  determine  how  these  approaches  may 
affect  organizational  effectiveness  with  a 
program  office. 

SUMMARY 

Many  real  world  problems  are  depicted  by  ooorly 
behaved,  multimodal  distributions  with  a  rela¬ 
tively  few  number  of  independent  variables 
(ouanti tati ve  and/or  qualitative).  This  unex¬ 
plained  variability  is  greatly  reduced  and  it 
becomes  unimodal  (and  falls  within  the  range 
of  skewness  and  kurtosis  tables)  when  properly 
included  in  the  mathematical  models.  Distri¬ 
butions  outside  the  range  of  the  tabled  values 
have  invariably  been  found  to  be  reducable  by 
the  introduction  of  an  additional  exolanatory 
variable  (Larew). 

Another  oanel  agreed  with  the  presen ta tor  that 
there  is  often  little,  if  any "justification 
for  the  use  of  uniform,  triangular,  trape¬ 
zoidal,  Weibull  or  PERT-type  distributions 
(they  are  all  typically  much  too  symmetrical 
and  too  flat  to  represent  real  life  data)." 

The  RS  distribution  is  by  far  superior  to  all 
other  percentile-type  distributions  for 
approximating  unimodal  random  variables 
( Larew) . 
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TAXONOMIC  CONSIDERATIONS  IN  RISK  AND  UNCERTAINTY 
by 

Robert  F.  Golden  and  Martin  D.  Martin 
Air  Force  Business  Research  Management  Center 


Abstract 

This  article  presents  a  number  of  ways  of 
constructing  a  taxonomy  of  risk  and  uncer¬ 
tainty  terms.  It  is  not  meant  to  be  prescrip¬ 
tive,  but  only  to  describe  ways  that  authors 
have  treated  risk  and  uncertainty.  Four  major 
classifications  are  described:  environmental, 
functional,  informational,  and  technical. 

From  this  description,  it  is  hoped  that  a  more 
conclusive  taxonomy  of  risk  arid  uncertainty 
can  be  developed. 


Definition  of  Risk  and  Uncertainty 

.In  its  simplest  form,  uncertainty  can  be 
defined  as  the  absence  of  information. 
Certainty,  conversely,  is  complete  or  total 
knowledge.  The  implications  for  the  decision 
maker  refer  to  his/her  ability  to  make  the 
correct  decision.  Depending  on  the  degree  of 
information  which  the  problem  solver 
possesses,  he  will  be  located  somewhere  along 
a  continuum  of  knowledge  which  can  be  called 
the  uncertainty  spectrum.  This  spectrum  is 
illustrated  in  Figure  1.  As  the  problem  is 
defined,  the  decision  maker  will  move  to  the 
left  on  this  spectrum. 


dealing  with  this  subject.  It  is  not 
intended  that  this  taxonomy  be  a  completely 
valid  and  exclusive  categorization,  but  that 
it  represents  the  majority  of  significant  work 
written  on  the  subject.  This  survey  is  pre¬ 
sented  as  a  starting  point  for  further  refine¬ 
ment  of  a  risk/uncertainty  taxonomy.  It  is 
hoped  to  stimulate  further  thinking  and 
discussion  in  the  workshop  on  this  subject. 

The  assumption  is  made  that  uncertainty 
and  risk  can  be  treated  as  synonomous  terms. 

A  review  of  the  uncertainty  spectrum  confirms 
the  difference  between  the  terms  as  one  of 
degree.  Decision  makers  usually  have  some 
information  to  serve  as  a  basis  for  decision 
making.  Seldom  will  a  decision  be  made  when 
no  information  is  available.  Rather  the  deci¬ 
sion  will  be  delayed,  and  action  initiated  to 
obtain  some  information.  Risk  implies  that 
the  probability  of  future  events  is  known  in 
terms  of  a  probability  distribution  (10). 
Uncertainty  generally  is  defined  as  a 
situation  where  the  probability  of  events  is 
not  known.  From  a  subjective  standpoint,  the 
decision  maker  usually  has  intuitive  feelings 
about  the  future  and  is  able  to  structure  a 
probability  distribution.  This  subjective 
estimation  of  probabilities  will  move  the 
decision  maker  into  the  risk  segment  of  the 
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Figure  1  -  The  Uncertainty  and  Information  Spectrum 


The  objective  of  this  article  is  to 
attempt  to  construct  a  taxonomy  of  uncertainty 
and  risk  based  on  a  survey  of  the  literature 


spectrum.  A  careful  definition  of  the  problem 
in  terms  of  past  ones  of  a  similar  nature  will 
assist  in  enhancing  the  visibility  of  sub¬ 
consciously  submerged  information. 
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Chance  (3),  Grayson  (6),  Bierman  and  Smidt 
(1),  Robichek  and  Myers  (15),  and  Lerner  and 
Carleton  (11)  support  this  assumption.  Knight 
maintains  that  the  two  concepts  should  be 
treated  as  separate  entities  (10).  Hunt  sup¬ 
ports  him  (8).  Hunt's  assertion  is  based  on 
Dean's  treatment  of  the  concepts  (4).  Dean 
defines  the  terms  and  treats  them  separately. 
Hwang  in  his  studies  takes  this  same  approach 
(9).  The  latter  three  individuals  concentrate 
their  efforts  in  the  risk  part  of  the  uncer¬ 
tainty  spectrum.  The  approach  taken  by  both 
groups  to  cope  with  the  unpredictability  of 
the  future,  however,  from  a  conceptual  stand¬ 
point  is  basically  the  same.  Therefore, 
uncertainty  will  be  equated  with  risk,  since 
the  terms  are  essentially  indistinguishable. 


TAXONOMIC  CLASSIFICATIONS 

A  survey  of  the  literature  reveals  that 
there  are  various  terms  which  have  been  used 
to  categorize  and  describe  uncertainty  or  risk 
factors.  In  this  article,  the  various  terms 
have  been  grouped  into  four  taxonomic  classes: 
environmental,  functional,  informational,  and 
technical.  These  classes  are  explained  in  the 
following  subsection. 

Environmental  Uncertainty 

The  first  class  of  uncertainty  is  environ¬ 
mental  uncertainty.  This  type  of  uncertainty 
relates  to  the  surrounding  conditions, 
influences  or  forces  that  affect  the  problem, 
project  or  situation.  Environmental  uncer¬ 
tainties  include  all  factors  bearing  on  the 
situation  both  within  and  without  the  project. 
Within  the  environmental  class,  there  are 
several  subclasses  which  will  be  discussed 
individually.  The  first  subclass  which  has 
been  identified  includes  the  four  categories 
listed  below: 

1.  Nature  -  The  uncertainty  is  related  to 
natural  factors,  such  as  storms,  floods, 
earthquakes,  climactic  conditions  and  acts  of 
God  (7). 

2.  Social  and  Political  -  This  term  relates 
to  the  impossibility  of  being  able  to  predict 
with  any  precision  the  actions  of  social  and 
political  groups  or  the  effects  of  social  and 
political  influences. 

3.  Communication  Media  -  The  disparities  that 
exist  in  the  access  which  people  have  to  the 
various  informational  media.  The  differences 
result  in  ignorance  on  the  part  of  many  groups 
and  individuals. 

4.  Time  -  The  passage  of  time  results  in 
changes  which  can  distort  the  results  of  deci¬ 
sions  based  on  a  past  state-of-aff airs. 


The  second  subclass  of  environmental 
uncertainties  addresses  the  factors  internal 
and  external  to  the  project.  They  are  defined 
as  follows: 

1.  External  -  Those  uncertainties  which 
relate  to  factors  external  to  a  project  which 
can  impinge  on  final  results  (13). 

2.  Internal  -  Internal  uncertainties  comprise 
those  stenming  from  the  technical  approach 
taken,  internal  management  approach  or  other 
elements  inherent  in  the  project  itself. 

The  third  subclass  is  similar  to  the 
second;  however,  it  approaches  uncertainty 
from  a  systems  perspective.  It  will  consider 
the  influences  of  al 1  the  projects  within  an 
organization  on  the  specific  project,  thus  the 
systems  perspective.  This  subclass  has  two 
elements:  exogenous  and  endogenous. 

Exogenous  factors  refer  to  the  stimulus, 
initiating  a  given  change,  which  comes  from 
outside  the  organization.  Endogenous  factors 
refer  to  the  stimulus,  initiating  the  change 
which  originates  within  the  organization. 

Functional  Uncertainty 

Functional  Uncertainty  is  the  second  major 
class  of  uncertainty.  In  this  context,  it 
refers  to  the  broad  functional  areas  which  are 
inherent  in  a  major  project.  The  four  cate¬ 
gories  of  Functional  Uncertainty  are  business 
risk,  financial  risk,  technological  risk,  and 
production  risk  which  are  defined  below: 

1.  Business  Risk  refers  to  the  uncertainty 
that  a  firm  has  about  its  future  income 
stream.  The  risk  is  associated  with  the 
firm's  operation  (17). 

2.  Financial  Risk  is  the  uncertainty  that  is 
generated  by  the  ratio  of  debt  to  equity  in 
the  capital  structure  or  the  amount  of  earn¬ 
ings  available  to  common  stockholders.  For 
contracting  the  risk  of  profit  or  loss  on  an 
individual  contract  is  involved  (16). 

3.  Technological  Risk  refers  to  the  changes 
in  the  state-of-the-art  that  can  render  a 
system  or  product  obsolete.  Thus,  uncertainty 
exists  as  to  how  long  an  item  can  remain  in 
the  operational  inventory  (14). 

4.  Production  Risk  refers  to  the  uncertainty 
resulting  from  the  assembly,  manufacture  or 
integration  of  component  parts.  Should  a  part 
not  be  available,  or  other  production  problems 
arise,  then  the  finished  product,  construction 
project,  etc.,  cannot  be  ready  on  time  and  its 
cost  can  be  affected  (12). 
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Informational  Uncertainty 

Informational  Uncertainty  is  the  third 
class  of  uncertainty  that  has  been  identified 
in  the  literature.  The  context  of  this  type 
of  uncertainty  centers  on  the  degree  of  aware¬ 
ness  of  decision  makers  of  the  relevant  infor¬ 
mation  that  impacts  the  situation.  Drake 
mentions  one  subclass  of  uncertainty  which  is 
anticipated  unknowns  and  unanticipated 
unknowns  (5).  He  observes  these  classifica¬ 
tions  in  the  environment  of  contractor  organi¬ 
zations  performing  projects  or  programs  under 
contract  to  a  customer,  as  in  the  situation  of 
a  Department  of  Defense  (DOD)  acquisition 
organization  charged  with  managing  the  devel¬ 
opment  and  production  of  a  new  weapons  system. 
In  this  environment,  a  civilian  company  would 
normally  do  the  engineering,  development  and 
manufacture  of  the  system,  under  a  formal 
written  contract  with  the  DOD.  Within  this 
subclass,  anticipated  unknowns  (5),  are  those 
unknowns  that  the  decision  maker  is  aware  of. 
He  realizes  that  there  will  be  a  problem  and 
anticipates  it.  In  these  situations  the  deci¬ 
sion  maker  is  not  caught  completely  unaware  of 
the  problem.  The  converse  of  the  anticipated 
is  the  unanticipated  unknowns.  This  refers  to 
those  unknowns  that  the  decision  maker  does 
not  forsee.  When  they  do  occur,  it  takes  the 
manager  completely  by  surprise.  Within  this 
subclass,  the  unknowns  under  consideration  are 
only  the  ones  that  affect  the  program  or  pro¬ 
ject  at  hand. 

A  separate  category  of  informational 
uncertainties  are  discussed  by  Burnham,  which 
he  calls  the  known  unknowns  and  the  unknown 
unknowns  (2).  Unlike  the  previous  category, 
these  factors  include  all  such  uncertainty 
factors,  whether  they  directly  influence  the 
project  or  not.  The  known  unknowns  are  those 
facts  that  the  contractor  knows  that  he  does 
not  know.  Within  this  same  category  are  the 
unknown  unknowns.  These  are  the  factors  that 
the  decision  maker  does  not  anticipate  - 
factors  that  he  does  not  know  he  doesn't  know. 

Technical  Uncertainty 

The  fourth  category  of  uncertainty  is 
Technical  Uncertainty.  Unlike  the  other 
categories,  technical  uncertainty  addresses 
the  meaning  of  uncertainty  in  its  technical 
sense.  It  deals  with  a  more  rigorous  defini¬ 
tion  of  the  meaning  of  uncertainty,  certainty 
and  risk.  Within  the  technical  aspects  of 
uncertainty  two  constructs  can  exist.  The 
first  deals  with  the  three  concepts  of 
uncertainty,  risk,  and  certainty.  The  second 
construct  considers  uncertainty  as  being 
either  subjective  or  objective.  These  terms 
are  explained  below. 

From  a  technical  viewpoint,  there  are 
three  situations  which  deal  with  uncertainty. 


The  first  is  certainty  where  each  decision 
leads  to  a  predictable  outcome.  In  this 
situation,  the  decision  maker  has  no  doubt  as 
to  the  final  outcome  of  his  action.  The  next 
situation  is  called  uncertainty.  Uncertainty 
occurs  when  the  known  is  completely  dominated 
by  the  unknown.  Uncertainty  denotes  that  pro¬ 
bability  distribution  for  future  events  are 
not  known.  In  the  case  of  risk  (10),  a  spe¬ 
cial  case  of  uncertainty,  a  decision  will 
result  in  a  specific  number  of  well  defined 
alternatives.  The  totality  of  the  outcomes 
for  this  situation  can  be  described  by  a  pro¬ 
bability  distribution.  When  the  term 
"uncertainty"  as  defined  in  the  previous 
paragraph  is  considered,  probabilities  can  be 
either  objective  or  subjective.  For  subjec¬ 
tive  uncertainty,  the  probabilities  assigned 
to  an  event  are  based  solely  on  the  obser¬ 
vation  choice  (18).  In  the  other  situation, 
objective  uncertainty,  the  probabilities  are 
derived  by  specific  procedures  independent  of 
the  problem  being  confronted. 

Summary 

Upon  examining  the  various  definitional 
categories  described  in  the  previous  section, 
it  becomes  apparent  that  from  a  generic 
standpoint,  the  environmental,  informational, 
and  tecnical  taxonomic  classes  can  be 
combined.  It  is  also  seen  that  the  functional 
class  is  another  way  of  describing  the  infor¬ 
mational  class  of  variables. 

Uncertainty  relates  to  events  which  will 
happen  in  the  future,  and  it  is  in  this  con¬ 
text  that  a  definition  takes  on  its 
importance.  The  challenge  for  the  decision 
maker  is  to  identify  and  understand  uncer¬ 
tainty  factors  as  they  relate  to  future 
problems.  The  decision  maker's  most  signifi¬ 
cant  task  is  to  identify  those  factors  which 
have  uncertain  aspects,  and  attempt  to  cope 
with  the  scarce  or  incomplete  information 
which  is  available  about  those  factors.  By 
developing  an  improved  taxonomy  of  uncertainty 
factors,  the  decision  maker  should  be  better 
able  to  both  identify  where  uncertainty 
exists,  and  cope  with  it. 
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A  DIMENSIONAL  APPROACH  TO  UNCERTAINTY  IN 
THE  SYSTEMS  ACQUISITION  PROCESS 

Charles  W.  N.  Thompson 
Northwestern  University 


ABSTRACT 


For  the  purpose  of  identifying  the  "key  parts 
of  (risk  and)  uncertainty  relating  to  (the 
systems  acquisition  process)"  from  the  view¬ 
point  of  the  (program)  manager,  it  is  proposed 
that  "the  process"  or  any  part  of  interest  be 
described  in  terms  of  one  or  more  ordinal  di¬ 
mensions.  As  one  approach  (and,  it  is  pro¬ 
posed,  a  preferred  approach),  dimensions  are 
described  with  respect  to  three  "parts"  of  the 
process:  the  plan,  the  program,  and  the  rela¬ 

tionship  of  the  program  manager  to  others. 

To  provide  a  context  for  this  use  of  a  dimen¬ 
sional  approach,  a  brief  discussion  is  pre¬ 
sented  of  other  dimensions  related  more  gener¬ 
ally  to  uncertainty,  together  with  appendices 
describing  the  dimensions  and  the  notation. 
Because  the  purpose  of  the  paper  is  to  provide 
a  basis  for  discussion  rather  than  agreement, 
it  is  presented,  admittedly,  in  "rough"  form 
and  without  explicit  citation  to  the  relevant 
literature . 


SECTION  I 


INTRODUCTION 


"Winning  is  never  having  to  say, 

'If  (Quote  from  Jesse  Thompson 
on  the  television  program  "NFL  Today," 
November  27,  1980. 


A.  BASIS  FOR  THIS  PAPER 


This  paper  was  written  in  response  to  an  invi¬ 
tation  to  participate  in  a  workshop  session 
which  would  examine  the  intersection  of  "un¬ 
certainty  , ""taxonomic  concepts,"  and  "major 
systems  acquisition."  These  three  areas  have 
been,  to  varying  degrees,  of  great  interest  to 
the  writer  over  a  period  of  about  twenty-five 
years,  and  ranging  from  the  relatively  ab¬ 
stract  to  very  pragmatic.  It  is  recognized 
that  each  of  these  areas  subtends  a  very  large 
set  of  disciplines  and  practices,  often  char¬ 
acterized  by  mutual  isolation  (and  disdain), 
requiring,  it  is  proposed,  a  posture  of  modest 
inquiry. 

The  remainder  of  this  introductory  section 
will  briefly  comment  on  the  focus  of  the  paper. 


Section  II  will  discuss  uncertainty  in  general; 
Section  III  will  present  a  brief  model  which 
will  be  used  to  introduce  the  main  argument 
which  appears  in  Section  IV.  Section  V  pro¬ 
vides  a  brief  summary. 


B.  RISK  AND  UNCERTAINTY 


The  terms  "risk"  and  "uncertainty"  are  asso¬ 
ciated  not  only  with  a  variety  of  commonplace 
meanings  but  also  with  extensive  formal  in¬ 
quiries  variously  identified  with  decision 
making,  the  area  of  probability  or  likelihood, 
utility  theory,  etc.  These  inquiries  range 
from  abstract  (and  mathematical)  to  pragmatic 
(and  experimental),  and  it  is  not  unlikely 
that  the  language  and  interests  of  one  inquirer 
may  not  even  intersect,  let  alone  agree,  with 
those  of  another.  The  present  paper  draws  upon 
some  modest  acquaintance  with  these  several 
fields  and  has  no  (present)  intention  of  show¬ 
ing  disrespect.  It  is  believed,  but  not  here 
proposed,  that  a  common  framework  can  be  des¬ 
cribed  which  would  account  for  (and,  perhaps, 
reconcile)  most,  if  not  all,  of  the  parts  of 
the  several  areas  of  inquiry. 

In  this  paper  risk  and  uncertainty  will  be  lim¬ 
ited  to  the  sense  of  "likelihood"  as  distin¬ 
guished  from  utility,  and  the  emphasis  will  be 
on  examining  the  former.  Similarly,  the  paper 
will  limit  itself  to  problem  identification 
rather  than  the  development  of  explicit  solu¬ 
tion  strategies  or  tactics.  It  should  be  clear 
that  this  choice  is  not  intended  to  indicate  a 
(present)  denigration  of  the  importance  of 
either  utility  or  of  approaches  to  solution. 


C.  WHY  USE  ORDINAL  DIMENSIONS? 


While  "taxonomy"  suggests  "the  systematic  dis¬ 
tinguishing,  ordering,  and  naming  of  type 
groups  within  a  subject  field,"  its  extensive 
development  and  application  in  several  fields 
implies,  at  least,  that  there  is  some  "order" 
to  be  found  or  that  it  would  be  "useful"  to 
impose  some  order.  Taxonomies  have  been 
"found"  or  "imposed"  in  the  natural  sciences 
and,  in  some  cases,  in  the  social  sciences, 
e.g.,  organization  theory.  A  taxonomy,  if  ap¬ 
propriate,  would  have  some,  or  all,  of  the  ad¬ 
vantages  and  uses  of  a  model.  It  is  here  pro¬ 
posed,  however,  that  a  taxonomy,  in  the  sense 
of  a  strictly  hierarchical  ordering  or  classi- 
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fication,  would  be  dysfunctional,  introducing 
more  error  and  confusion  than  order. 

It  is  not  argued  that  a  taxonomy  would  not  be 
"nice"  to  have.  The  power  and  rigor,  and 
ability  to  build  upon  the  prior  art,  which  a 
schema  of  a  set  of  clearly  described  and  ar¬ 
ticulated  concepts,  supported  with  quantitative 
measures  and  data,  provides  is  well  established 
in  science.  It  is,  however,  argued  that,  with 
the  possible  exception  of  theoretical  or  lim¬ 
ited  experimental  inquiries,  the  art  does  not 
yet  support  such  a  science.  If  this  is  true, 
it  is  proposed  that  the  (practical)  problem  of 
dealing  with  risk  and  uncertainty  be  approached 
through  use  of  a  series  of  "dimensions,"  dimen¬ 
sions  which  do  not  require  (nor  imply)  a  rig¬ 
orous  interdependence,  e.g.,  mutual  exclusive¬ 
ness  or  inclusiveness.  It  is  proposed  that  the 
set  of  dimensions  be  defined  in  name  and  form 
as  ordinal,  representing  a  series  of  viewpoints 
(or  views)  of  the  "problem,"  rather  than  a  for¬ 
mal  ordering  (see  Appendix  I). 

It  seems  clear  that  the  choice  of  form  or  di¬ 
mension  (or  scale)  is  properly  the  function  of 
the  data  and  what  one  wants  to  do  with  it.  A 
poet  might  well  prefer  a  nominal  scale  with 
its  freedom  in  choice  of  data  and  manipulation; 
a  physicist  dealing  with  complex  phenomena 
might  well  require  a  ratio  scale  to  take  advan¬ 
tage  of  the  "order"  in  the  phenomena  and  the 
convenience  in  using  advanced  mathematical 
analysis.  Only  where  the  "order"  is  there  in 
the  phenomena  and  the  need  for  analytical 
techniques  to  achieve  one’s  purpose  exists  is 
the  use  of  more  "ordered"  scales  clearly  ap¬ 
propriate.  Of  these  conditions,  it  is  pro¬ 
posed  that  only  the  second  is  presently  met. 

It  is  proposed  that  there  are  some  considerable 
advantages  in  using  an  ordinal  scale  for  exam¬ 
ining  uncertainty  in  relation  to  the  systems 
acquisition  process.  Ordinal  scales  are  con¬ 
venient,  they  are  easy  to  define  and  apply, 
they  tend  to  be  stable  over  a  set  of  problems 
(in  management),  and  they  are  easily  recognized 
as  "inappropriate"  (when  that  is  true)  and 
changed  to  a  more  "correct"  scale.  Most  of  the 
variables  ("things")  to  be  measured  appear  to 
take  on,  at  least,  ordinal  variations,  al¬ 
though,  in  some  cases,  such  as  "awareness,"  it 
may  appear  that  a  nominal  scale  is  sufficient. 
Changes  in  uncertainty  are  easily  described  in 
ordinal  terras,  as  are  "matches"  of  achieved 
levels  of  uncertainty  to  levels  "required"  for 
some  decision  process. 


D.  SYSTEMS  ACQUISITION 


The  relationship  of  uncertainty  to  systems 
acquisition  depends,  to  a  large  extent,  on  the 
viewpoint  of  the  individual  concerned.  In  this 
paper,  the  viewpoint  will  be  assumed  to  be  that 
of  the  program  manager  (in  the  government). 


There  are,  of  course,  other  important,  and  rel¬ 
evant,  viewpoints.  From  the  viewpoint  of  a 
(prospective)  contractor,  the  concern  may  vary 
considerably,  A  senior  executive  might  view 
risk  and  uncertainty  (almost)  solely  in  eco¬ 
nomic  terms:  what  is  the  expected  profit  or 
loss,  what  is  the  (likely)  effect  on  long  term 
growth  and  equity,  what  is  the  effect  on  li¬ 
quidity?  A  technical  executive  might  view  un¬ 
certainty  in  terms  of  technical  performance: 
what  is  the  likelihood  that  the  specifications 
can  be  met,  at  all,  within  budget  and  schedule? 
The  using  (or  operating)  organization  within 
the  government  may  well  have  different  priori¬ 
ties  in  viewing  uncertainty:  how  will  the 
system  perform,  will  it  meet  specifications, 
and  even  if  it  does,  will  it  do  the  job,  and 
will  it  be  available  on  schedule,  and  when  re¬ 
quired  during  its  time  in  the  inventory?  Pol¬ 
icy  level  executives  in  the  government  may  well 
be  concerned  with  all  of  the  above  viewpoints 
as  well  as  broad  uncertainties  in  the  system's 
relationship  with  other  competing,  or  inter¬ 
acting,  or  higher  level  systems. 


SECTION  II 


DIMENSIONS  OF  UNCERTAINTY  (IN  GENERAL) 


A.  UNCERTAINTY  (AND  CONFIDENCE) 


At  a  relatively  abstract  and  general  level,  the 
concept  of  ’’uncertainty"  would  appear  to  be 

reasonably  "certain"  -  it  seems  to  mean  "how 

certain  some  'thing'  is,"  and  can  be  stated  or 
described  in  terms  of  a  dimension  which  extends 
from  UNCERTAINTY  ----  CERTAINTY,  or  vice  versa 
(see  Appendix  II  for  a  more  complete  defini¬ 
tion).  While  there  is  a  considerable  litera¬ 
ture  describing  the  dimension  (and/or  its 
scale),  and  means  for  obtaining  measures,  it 
seems  clear  that,  for  at  least  most  purposes, 
the  concept  is  associated  with  (one  or  more) 
sentient  being(s),  and  with  respect  to  some  ob¬ 
ject,  event,  relationship,  etc.  Restated, 
"uncertainty"  is  a  perception  of  some  person 
(or  persons)  with  respect  to  some  relationship 
between  two  or  more  objects,  events,  relation¬ 
ships,  etc.  And,  for  at  least  some  purposes, 
the  concept  of  "confidence"  may  be  used,  re¬ 
ciprocally,  for  "uncertainty." 


B.  THE  DIMENSIONS  OF  UNCERTAINTY 


It  is  proposed  that  "changing  (one's)  uncer¬ 
tainty"  (or  changing  one's  confidence)  is  not 
only  ubiquitous  but  is  also  a  (the)  unique 
characteristic  of  sentient  (human)  beings.  The 
individual  achieves  (or  seeks  to  achieve)  some 
LEVEL  OF  CONFIDENCE  (Appendix  II)  with  respect 
to  what  he  sees,  or  feels,  or  perceives,  and 
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may  even  achieve  (or  seek  to  achieve),  with 
respect  to  that  confidence  (C),  some  level  of 
CONFIDENCE  IN  C.  While  this  concept  of  a 
change  in  confidence  is  proposed  as  a  ubiqui¬ 
tous  epistemological  process,  it  has  been  ex¬ 
amined  and  formalized,  with  a  considerable  de¬ 
gree  of  self-consciousness,  in  the  research 
(and/or  academic)  process.  The  "relationship" 
which  is  the  object  (or  subject)  of  uncertain¬ 
ty  (confidence)  is  conventionally  described  as 
an  "hypothesis,"  some  specified  relationship 
between  two  or  more  variables,  and  it  is  this 
relationship  which  is  the  focus  of  the  EFFECT 
UPON  UNCERTAINTY  (CONFIDENCE).  It  is  further 
proposed  that  the  reduction  of  uncertainty 
(increase  in  confidence)  is  achieved  only  by 
decreasing  the  relationship  of  "other"  vari¬ 
ables,  PARAMETERS,  with  the  variable  (s)  speci¬ 
fied  in  the  hypothesis.  There  are  many  di¬ 
mensions  which  might  be  used  to  describe  pa¬ 
rameters,  including,  for  example,  those  asso¬ 
ciated  with  isolating  the  hypothesized  phenom¬ 
enon  both  in  LENGTH  OR  DEPTH  and  in  BREADTH. 

The  literature  of  research  methodology  includes 
categories  such  as  endogenous  and  exogenous, 
experimental  threats,  experimenter  threats, 
etc . 


C.  SOME  SELECTED  DIMENSIONS  OF 
THE  UNCERTAINTY  REDUCTION  PROCESS 


The  research  process  may  be  considered  a  self- 
conscious  subset  of  the  more  general  epistemo¬ 
logical  process,  and  one  which  has  developed  a 
rich,  and  often  confusing,  language  to  describe 
the  various  parts  of  the  process. 

Certainly  early  in  the  process  is  acquiring 
some  awareness  of  the  focus  of  the  inquiry, 
the  "problem"  to  be  solved,  i.e.,  identifying 
the  relationship  for  which  one  wishes  to  in¬ 
crease  one's  confidence  (to  reduce  one's  un¬ 
certainty).  Identifying  the  "problem"  might 
be  characterized  by  a  number  of  dimensions, 
including  at  least  these: 

1.  How  well  can  the  problem  be  identi¬ 
fied,  i.e.,  to  what  degree  is  it 
UNSTRUCTURED  ----  STRUCTURED? 

2.  How  well  can  the  problem  be  isolated 

or  separated  from  other  problems  (or 
systems),  i.e,,  is  it  OPEN  - CLOSED? 

3.  Assuming  the  problem  is  big  enough 
or  important  enough  to  work  on, 

i.e.,  not  trivial,  is  it  small 
enough,  i.e.,  not  cosmic,  so  that, 
with  reasonable  effort,  one  can 
hope  to  "solve"  it,  i.e.,  is  it 
KNOWABLE? 

Assuming  the  researcher  has  been  able  to  iden¬ 
tify  his  problem,  and  assuming  it  is,  at  least, 
related  to  phenomena  in  the  real  world,  he  must 
consider  his  relation  to  the  problem,  i.e., 
how  he  will  go  about  solving  it,  including: 

1.  Where  is  the  "problem"  from  a 


time  point  of  view,  i.e,, 

RETROSPECTIVE  -  PROSPECTIVE? 

2.  Is  he  interested  in  exploring  the 
problem  (because  he  knows  little 
about  it)  or  is  he  interested  in 
testing  (or  proving)  a  solution 
he  is  pretty  sure  of,  i.e., 

EXPLORATORY  ----  A  PRIORI? 

3.  Is  he  just  going  to  observe  the 
phenomena  or  is  he  going  to  in¬ 
troduce  some  changes,  i.e., 

STUDY - EXPERIMENT? 

4.  Will  he  make  direct  measurements 
or  will  he  use  secondary  sources 

of  data,  i.e.,  NORMATIVE  -  EMPIRICAL? 

Given  that  the  researcher  has  been  able  to 
identify  his  problem  and  decided  how  to  ap¬ 
proach  it,  he,  normally,  will  gather  data  and 
analyse  it.  This,  again,  is  a  multi-dimen¬ 
sional  process,  including: 

1.  Is  he  looking  for  facts  or  opinions, 

i.e.,  FACT  - VALUE? 

2.  And,  in  either  case,  where  will 
he  get  them,  i.e., 

SUBJECTIVE  ----  OBJECTIVE? 

3.  And  will  the  measures  he  uses 

(and  the  phenomena  he  measures) 
facilitate  his  analytical  process, 
i.e.,  NOMINAL - RATIO? 

4.  And  will  he  be  able  to  use  dif¬ 
ferent  measuring  techniques,  i.e., 
VALIDATION  -  MULTI-METHOD? 

5.  And  will  he  be  able  to  get  dif¬ 
ferent  measures,  i.e., 

VALIDATION  -  MULT I -TRAIT? 


D.  A  PARTIAL  VIEW  OF  THE  DIMENSIONS 
OF  (UNCERTAINTY  WITH  RESPECT  TO)  UTILITY 

While  it  may  be  argued  that  uncertainty  (or 
confidence)  is  not  properly  considered  without 
also  considering  utility,  it  is  proposed,  for 
purposes  of  this  paper,  to  assume  that  there  is 
something  called  "utility"  which  bears  some  re¬ 
lation  to  uncertainty  but  to  treat  it  only  at 
the  nominal  level. 

Nominally,  in  parallel  with  the  dimensions  of 
uncertainty  discussed  in  B.,  above,  the  indi¬ 
vidual  will  be  concerned  with  LEVEL  OF  CONFI¬ 
DENCE  (WITH  RESPECT  TO  UTILITY).  While  it  ap¬ 
pears  clear  that  the  "practical  man,"  i.e.,  the 
program  manager,  will  be  concerned  with  util¬ 
ity,  it  should  also  be  made  clear  that  the  re¬ 
searcher,  whatever  his  preoccupation  with 
"confidence,"  is  also  concerned  with  utility, 
at  least  at  the  nominal  level.  It  is  proposed 
that  it  would  be  desirable  to  separate  the  two 
concerns  analytically  by  interposing  a  separ¬ 
ate  set  of  variables  between  utility  and  the 
variables  of  the  "hypothesis"  when  the  concern 
is  the  EFFECT  UPON  UTILITY  (PURPOSE,  GOALS, 
ETC.).  For  this  purpose,  it  is  proposed  that 
three  kinds  of  variables  will  be  sufficient: 
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1.  Variables  which  account  for  "cost," 
including  "time,"  i.e.,  RESOURCES /CON¬ 
STRAINTS 

2.  A  variable  (or  variables)  which 
account (s)  for  the  desired  (or 
proposed,  or  expected)  "result" 
of  the  hypothesis,  i.e.,  SUBSE¬ 
QUENT  EFFECT 

3.  Variables  which  account  for  "other 
effects,"  i.e.,  SPINOFF. 


E.  SOME  SELECTED  DIMENSIONS 
OF  THE  DECISION  MAKER 


The  above  discussion  provides  a  general  des¬ 
cription  of  the  epistemological  process,  ap¬ 
plicable  without  distinction  across  all  deci¬ 
sion  makers.  However,  the  program  manager  is 
a  specific  decision  maker,  with  specific  char¬ 
acteristics  and  dealing  with  specific  deci¬ 
sions.  His  interest,  to  apply  the  above,  is 
in  the  relationship  of  uncertainty  to  some  de¬ 
cision  he  will  (may)  make,  i.e.,  LEVEL  OF  CON¬ 
FIDENCE  (DECISION  MAKING),  and  LEVEL  OF  CONFI¬ 
DENCE  WITH  RESPECT  TO  UTILITY  (DECISION  MAK¬ 
ING). 

As  a  decision  maker  "embedded"  in  the  real 
world,  he  must  determine  how  much  freedom  he 
has  to  make  a  (the)  decision,  i.e.,  his  AU¬ 
THORITY/INDEPENDENCE  (WITH  RESPECT  TO  UNCER¬ 
TAINTY),  and  (WITH  RESPECT  TO  UTILITY).  And, 
even  if  he  has  the  necessary  "freedom"  or 
choice,  with  respect  to  others,  he  himself  may 
be  limited  by  his  own  characteristics,  in¬ 
cluding  the  following: 

1.  KNOWLEDCE,  SKILLS  AND  PROBLEM 
SOLVINC  ABILITY 

2.  TOLERANCE  OF  AMBICUITY 

3.  NEED  TO  STRUCTURE 

4.  RISK  AVERSION. 

Given  that  all  of  the  above  dimensions  "match" 
appropriately  with  the  decision  he  is  consid¬ 
ering,  he  must  then  consider  the  uncertainty 
(and  utility)  related  substantively  to  the  de¬ 
cision  itself,  in  the  form  of  the  following: 

1.  FEASIBILITY  (CREDIBILITY) 

2.  TIMELINESS 

3.  COSTLINESS. 


F .  COMMENTS 


These  general  dimensions  are  not  now  proposed 
to  be  complete  in  terms  of  coverage  or  in  terms 
of  exposition.  They  are  presented  solely  to 
provide  a  context  for  considering  the  dimen¬ 
sions  to  be  presented  in  Section  IV,  below. 


I 

SECTION  III 


IDEALIZED  PROGRAM  MANACER'S  POINT  OF  VIEW 
(IN  THE  ABSENCE  OF  UNCERTAINTY) 


A  world  without  uncertainty  may  provide  a  use¬ 
ful  simplification,  somewhat  analogous  to  a 
frictionless  mechanical  model.  The  functions, 
activities,  etc.,  of  a  program  manager  might  be 
described  in  relatively  simple  terms.  First, 
he  is  assigned  or  given  responsibility  (and 
authority)  to  do  all  of  the  things  necessary  to 
carry  out  the  program.  The  program  is  des¬ 
cribed  to  him  (is  given  to  him)  in  the  form  of 
a  program  plan.  The  plan  describes  the  pro¬ 
gram  (at  least)  in  terms  of  its  beginning  or 
initiation  and  its  completion  or  end  point.  As 
a  minimum,  the  plan  specifies  the  end  point  in 
terms  of  three  dimensions: 

1.  performance  -  the  desired  char¬ 
acteristics  or  capabilities  of 
the  system  or  equipment  which 
is  planned  or  desired 

2.  schedule  -  the  planned  or  de¬ 
sired  time  at  which  the  system 
will  be  in  existence  or  available 

3.  cost  -  the  planned  or  desired 
amount  and  type  of  resources  which 
will  be  expended  or  used  in  ob¬ 
taining  the  system. 

The  description  of  the  desired  performance 
might  include  not  only  its  nominal  capability 
(i.e.,  technical  equipment  specification)  but 
also  other  desired  characteristics  (e.g.,  how 
long  it  will  perform,  under  what  range  of  en¬ 
vironmental  conditions,  how  it  will  interface 
with  other  systems).  In  addition,  information 
describing  the  above  may  be  desired  in  the  form 
of  data  to  allow  other  systems  to  interact  with 
it  (e.g.,  training,  operating,  installing,  and 
maintenance  instructions). 

The  description  of  the  desired  schedule  might 
include  not  only  the  desired  end  delivery 
schedule  but  also  the  desired  schedule  for  in¬ 
termediate  products  (e.g.,  information  on  tech¬ 
nical  interfaces)  or  intermediate  inputs  (e.g., 
information  on  technical  interfaces). 

The  description  of  cost  might  include  not  only 
a  total  (dollar)  budget  but  also  specification 
of  particular  resources  (e.g.,  personnel,  fa¬ 
cilities,  contractors,  etc.) 

In  classical  management  terms,  the  program  man¬ 
ager  will  then  do  the  following; 

1.  planning  -  break  down  the  program 
plan  into  a  set  of  smaller  plans 
(sub-plans),  interrelated  in  se¬ 
quence  and/or  in  parallel 

2.  organizing  -  establish  the  form  of 
organization  which  will  be  used  to 
carry  out  the  plan 
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3.  staffing  -  obtain  and  assign 
specific  individuals  to  the 
organization 

4.  directing  -  provide  the  plan  to 
the  staff 

5.  controlling  -  observe  (from  time 
to  time)  the  actual  dimensions  of 
the  program  (or  its  parts),  compare 
with  the  plan,  and  provide  direc¬ 
tion  to  bring  the  actual  into  agree¬ 
ment  with  the  plan. 

While  the  absence  of  uncertainty  would  suggest 
that  the  process  might  be  even  simpler  (e.g,, 
there  would  be  no  need  to  control),  this  pro¬ 
cess  will,  at  the  end  point,  result  in  an  ac¬ 
tual  system  which  agrees  with  the  planned  per¬ 
formance  description,  and  at  the  scheduled 
(planned)  time  and  planned  cost. 

Unfortunately,  this  scenario  seldom,  if  ever, 
occurs  in  programs  of  any  size  or  complexity. 
Either  the  actual  system  does  not  agree  with 
the  program  plan,  or  the  program  plan  changes 
during  the  course  of  the  program,  or  both,  and 

usually  in  ways  which  are  not  desired  -  less 

performance,  more  time,  more  cost. 


SECTION  IV 


PROGRAM  MANAGER’S  POINT  OF  VIEW 
(INCLUDING  UNCERTAINTY) 


A.  INTRODUCTION 


It  may  be  useful  to  introduce  uncertainty  in 
terms  of  the  idealized  model  presented  above, 
recognizing  that  the  idealized  model  is  an  ab¬ 
straction  from  practice  or  conventional  wisdom 
which  itself  operates  in  the  presence  of  un¬ 
certainty.  If  it  can  be  assumed  that  practice 
is,  itself,  evidence  (if  imperfect)  of  a  (or 
the)  way  to  model  the  real  world,  it  may  be 
useful  to  comment  on  how  this  practice  relates 
to  uncertainty. 

The  idealized  model  includes  at  least  two  ma¬ 
jor  elements:  a  PLAN  and  a  PROGRAM.  In  the 
systems  literature  (and  practice)  we  find  a 
similar  division:  planning  and  execution.  In 
the  management  literature  (and  practice)  we 
find  a  similar  division:  planning  and  organiz¬ 
ing/staffing/directing/controlling.  It  is  sug¬ 
gested  that  this  division  is  a  common  thread, 
ubiquitous  in  a  variety  of  forms  throughout  the 
prior  art  and  practice.  In  terms  of  uncertain¬ 
ty  (and  utility),  it  is  proposed  that  this  di¬ 
vision  provides  a  powerful  (and  useful)  basis 
for  modeling  in  the  following  way.  The  func¬ 
tion  of  the  plan  is  to  provide  a  '’boundary" 
between  the  uncertainties  (and  utilities)  of 
the  rest  of  the  real  world  and  that  part  of  it 
which  is  the  program.  It  provides  (or  is  in¬ 


tended  to  provide)  some  nominal  level  of  cer¬ 
tainty  (and  utility)  sufficient  to  (make  deci¬ 
sions  to)  carry  out  the  program.  Within  this 
closed  system,  the  uncertainties  (and  utili¬ 
ties)  are  limited  to  those  with  respect  to  re¬ 
lationships  within  the  program,  and  with  re¬ 
spect  to  relationships  between  the  program  and 
the  plan. 

Introducing  uncertainty  into  the  idealized 
model  suggests  at  least  two  critical  dimen- 
s ions : 

1.  Uncertainty  with  respect  to  the 
PLAN 

2.  Uncertainty  with  respect  to  the 
PROGRAM  (both  within  the  PRO¬ 
GRAM  and  in  relation  to  the  PLAN). 

These  uncertainties,  however,  introduce  another 
complication  in  terms  of  decision  making  (and 
decision  makers).  In  the  idealized  model,  the 
PLAN  is  the  concern  of  decision  makers  outside 
of  the  PROGRAM,  and  the  PROGRAM  is  the  concern 
of  decision  makers  within  the  PROGRAM  (here, 
the  program  manager  and  those  he  represents). 
The  uncertainties  with  respect  to  the  plan  and 
the  program,  in  practice,  become  the  concern 
(usually)  of  both  sets  of  decision  makers,  re¬ 
sulting  in  uncertainties  in  the  relationships 
between  (at  least)  the  two  sets  of  decision 
makers.  This  suggests  the  addition  of  a  third 
dimension : 

3.  Uncertainty  with  respect  to  the 
(authority)  relationships. 

It  is  proposed  that  these  three  dimensions  pro¬ 
vide  a  useful  base  or  framework  for  the  key 
parts  or  elements  of  uncertainty  (and  utility) 
from  the  program  manager's  point  of  view. 

While  the  above  is  proposed  as  the  major  frame¬ 
work,  it  should  be  clear  that  there  are  others 
which  focus  on  other  dimensions,  e.g.,  the 
steps  or  phases  in  the  acquisition  cycle. 

The  dimensions  described  above  will  provide  the 
outline  for  the  sections  which  follow. 


B.  UNCERTAINTY  WITH  RESPECT  TO  THE 
(PROGRAM)  PLAN 


Identifying  uncertainties  with  respect  to  the 
PLAN  may  start  by  focusing  on  the  relation  of 
the  PLAN  to  the  program  manager.  First  of  all, 
the  PLAN  proposes  to  describe  three  interre¬ 
lated  changes  which  will  (can,  should)  occur 
during  the  period  of  the  program,  as  follows: 

1.  A  change  in  performance  from  some 
(usually)  unstated  level  of  per¬ 
formance  at  to  some  specified 
level  of  performance  at  te 

2.  A  change  in  time  from  t^  to  te, 
usually  stated  only  in  terms  of  te 

3.  A  change  in  "cost"  or  resources 
from  some  initial  specified  level 
to  zero  (or  from  zero  to  some 
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specified  end  level) . 

The  PLAN  assumes  (or  proposes)  that  these  three 
specified  changes  are  related  in  such  form  that 
they  will  all  occur  during  the  program,  but 
this  may  not  be  certain,  i.e.,  it  may  cost  more 
(or  less)  and  take  more  (or  less)  time  to 
achieve  a  higher  (or  lower)  level  of  perform¬ 
ance.  This  "uncertainty"  may  be  described  as 
the  "feasibility"  of  the  PLAN,  and  provides  the 
first  dimension: 

PUN  FEASIBILITY 

The  degree  to  which  the  plan  (perform¬ 
ance,  schedule,  and  cost;  or  the  combin¬ 
ation  or  interrelation)  can  be  (or  is) 
in  agreement  with  (or  performance  is 
equal  to  or  greater  than,  and  schedule 
and  cost  is  equal  to  or  less  than)  the 
state  (achieved,  actual,  measured)  at 
the  time  of  completion  (te)  of  the 
planned  activity  (PROGRAM). 

In  a  conventional  sense,  this  dimension  des¬ 
cribes  the  primary  question  a  program  manager 
is  faced  with:  "Can  I  (or  anyone)  deliver 
what  they  want  with  the  amount  of  time  and 
money  they've  given  me?" 

The  second  major  dimension  of  uncertainty  with 
respect  to  the  PUN  arises  out  of  its  major 

purpose  -  to  provide  the  program  manager  with 

a  statement  of  his  (and  the  program's)  relation 
to  the  world  outside  of  the  program.  The  PUN 
proposes  that  this  combination  of  performance/ 
cost/schedule  will  meet  or  exceed  some  level  of 
utility  (or  some  level  of  uncertainty  with  re¬ 
spect  to  utility).  While  this  may  not,  nomin¬ 
ally,  be  a  concern  of  the  program  manager,  as 
a  practical  matter  it  does,  and  may  be  des¬ 
cribed  as  the  "utility"  of  the  PLAN,  and  pro¬ 
vides  the  second  dimension: 

PLAN  UTILITY 

The  degree  to  which  the  plan  agrees  with 
(or  exceeds)  some  preferred  external 
state  (i.e.,  some  preference  standard, 
or  some  preferred  state  with  relation  to 
some  external  plan). 

In  a  conventional  sense,  this  dimension  des¬ 
cribes  the  secondary  question  a  program  manager 
is  faced  with:  "If  I  deliver  what  they  want 
(what  the  plan  says  they  want)  on  time  and  on 
schedule,  will  the  program  be  a  'success'?" 

In  addition  to  these  two  basic  dimensions,  a 
number  of  other  dimensions  may  describe  speci¬ 
fied  uncertainties.  First,  plans  are  imperfect 
descriptions,  i.e.,  vary  in  "clarity": 

PUN  CURITY 

The  degree  to  which  the  description  of 
the  plan  agrees  with  the  PUN. 

Second,  the  PUN  as  initially  stated  may  change 
by  the  end  of  the  program,  i.e.,  may  lack 
"stability": 

PUN  STABILITY 

The  degree  to  which  the  initial  PUN 
(at  ti)  is  in  agreement  with  the  final 
PUN  (at  t  ). 

Third,  the  PL^N  may  change  one  or  more  times 
during  the  program  (as  distinguished  from 


the  above),  i.e.,  it  may  have  "variability": 
PUN  VARIABILITY 

The  degree  to  which  the  initial  PUN  (at 
t^)  is  in  agreement  with  the  PUN  at  tn 
(where  ta  is  tj>tn>te;  and  tn  may  vary  in 
timing  and  frequency,  i.e.,  the  fre¬ 
quency  and  distribution  of  the  set  of  tn) . 
Fourth,  the  plan  may  be  described  in  terms  of 
nominal  states,  e.g.,  delivery  scheduled  for 
February  10,  1982,  which  may  not  agree  with  the 
range  of  preferred  states,  i.e.,  may  or  may  not 
include  "flexibility": 

PUN  FLEXIBILITY 

The  degree  to  which  the  PUN  specifies 
a  range  of  preferred  states  (or  com¬ 
binations)  at  t«. 

m 

Fifth,  the  PUN  may  include  a  number  of  inter¬ 
related  details  which  are  assumed  to  be  con¬ 
sistent,  i.e.,  introduces  "complexity": 

PUN  COMPLEXITY 

The  degree  to  which  the  plan  includes  In¬ 
termediate  (sequential)  and/or  detail 
(parallel)  states  which  may  affect 
feasibility. 

Sixth,  a  variation  on  the  above  occurs  in  what 
is  known  as  "process"  versus  "product"  speci¬ 
fication: 

PUN  SPECIFICATION  (PROCESS /PRODUCT) 

The  degree  to  which  the  plan  is  limited 
to  a  description  of  the  state  at  te 
(product)  as  distinguished  from  a  des¬ 
cription  of  states  at  tn  (process). 

(Note:  a  PUN  may  include  both,  which 
introduces  feasibility  issues;  and  a 
plan  may  include  neither,  which  intro¬ 
duces  flexibility  issues). 

These  dimensions,  with  the  possible  exception 
of  the  first  two,  may  be  considered  to  repre¬ 
sent  a  selection  of  the  salient  dimensions  of 
uncertainty  with  respect  to  the  plan.  It  is 
suggested,  however,  that  they  do  represent  a 
reasonable  starting  point. 


C.  UNCERTAINTY  WITH  RESPECT  TO  THE  PROGRAM 


The  uncertainties  with  respect  to  the  program 
are,  particularly  in  research  and  development, 
the  major  focus  of  a  program  manager.  Even  as¬ 
suming  the  PUN  is  feasible,  the  program  man¬ 
ager  must  select  and  carry  out  a  set  of  deci¬ 
sions  which  reflect  that  feasibility.  For  pur¬ 
poses  of  this  paper,  the  uncertainties  here 
will  be  limited  to  those  which  are  summarized 
in  the  classical  management  functions:  plan¬ 
ning,  organizing,  staffing,  directing,  and  con¬ 
trolling. 

The  first  of  these,  planning,  is,  in  one  sense, 
accomplished  by  the  PUN.  However,  in  most 
programs,  the  program  manager  will  find  it  de¬ 
sirable,  if  not  necessary,  to  break  up  the 
program  into  a  series  of  sub-programs.  This 
planning  function  introduces  a  number  of  un¬ 
certainties,  but,  as  a  "plan,"  these  uncer- 
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tainties,  at  a  different  level,  will  be  simi¬ 
lar  if  not  identical  with  those  already  des¬ 
cribed  . 

While  for  each  of  the  other  four  functions  a 
comparable  set  of  dimensions  with  reference  to 
uncertainty  could  be  developed,  for  present 
purposes  a  brief  summary  for  each  will  be  made: 
PROGRAM  MANAGEMENT  (ORGANIZING) 

The  degree  to  which  the  specified  sub¬ 
programs  and  their  relationships  agree 
with  the  program  plan 
PROGRAM  MANAGEMENT  (STAFFING) 

The  degree  to  which  the  specified  (per¬ 
sonnel)  resources  agree  with  the  pro¬ 
gram  plan 

PROGRAM  MANAGEMENT  (DIRECTING) 

The  degree  to  which  the  specified  in¬ 
formation  about  the  plan  and  its  com¬ 
munication  to  staff  agree  with  the 
program  plan 

PROGRAM  MANAGEMENT  (CONTROLLING) 

The  degree  to  which  specified  infor¬ 
mation  about  variation  between  the  plan 
and  sub-plans  agrees  with  the  program  plan. 

The  above  set  of  dimensions  is  only  one  of 
several  general  sets  of  dimensions.  For  ex¬ 
ample,  the  program  manager  might  limit  himself 
to  the  two  dimensions  of  "ef fectiveness"  and 
"ef f iciency ,"  with  various  definitions  and, 
perhaps,  a  set  of  sub-dimensions.  Another  set, 
still  at  the  general  level,  might  Include  the 
dimensions  of  "understanding  the  problem," 
"present  or  proposed  approach,"  and  "present 
or  potential  capability  to  solve  the  problem." 
Each  of  these  sets,  Including  the  set  of  five 
functions,  might  be  applied  to  the  program  as 
a  whole,  or  to  any  sub-part. 

At  a  somewhat  less  general  level,  sets  of  di¬ 
mensions  might  be  identified  with  specific 
sub-parts  of  the  program.  Particular  technical 
problems,  or  the  set  of  specific  technical 
problems,  are  easily  associated  with  uncer¬ 
tainty.  Similarly,  problems  related  to  cost 
or  schedule  may  be  identified.  Alternative 
specific  sets  may  be  based  upon  any  method 
used  for  identifying  sub-parts  of  the  program. 


D.  UNCERTAINTY  WITH  RESPECT 
TO  PROGRAM  AUTHORITY 


In  the  absence  of  uncertainty,  a  PLAN  might  de¬ 
termine  the  "authority"  relationship,  i.e., 
establish  the  bounds  of  decentralization,  the 
limits  on  discretion  or  choice  of  the  program 
manager,  with  respect  to  execution  of  the  pro¬ 
gram,  I.e.,  program  management.  However,  with 
uncertainty  in  plan  and  program,  there  may  also 
be  uncertainty  in  the  relationship  between 
those  outside  the  program  and  the  program 
manager.  During  the  period  of  the  program 
those  outside  of  the  program  may  wish  to  pro¬ 
vide  Input  or  obtain  output  concerning  Inter¬ 


nal  program  management  decisions;  and  the  pro¬ 
gram  manager  may  wish  to  "cross  the  boundary" 
in  the  other  direction  with  respect  to  deci¬ 
sions  affecting  the  PLAN.  While  the  uncer¬ 
tainties  here  might  be  subsumed  under  the 
PLAN  dimensions,  it  is  proposed  that  they  are 
sufficiently  important,  and  different,  to  war¬ 
rant  a  parallel  category.  In  general,  this 
dimension  relates  to  "program  authority"  and 
may  be  described  as  follows: 

PROGRAM  AUTHORITY 

The  degree  to  which  one's  decision  (on 
planning  and/or  execution)  is  not  depend¬ 
ent  upon  (affected  by  or  directed  by)  In¬ 
put  (present  or  future)  from  others 
And,  following  the  three  divisions  of  the  PLAN, 
three  component  dimensions  can  be  Identified 
as  follows: 

DESIGN  AUTHORITY 

The  degree  to  which  one's  decision  on 
choice  of  technical  means  (X^v)  and  re¬ 
sults  (Xcjv)  is  not  dependent  upon  (af¬ 
fected  by  or  directed  by)  input  (present 
or  future)  from  others 
SCHEDULING  AUTHORITY 

The  degree  to  which  one's  decision  on 
choice  of  when  a  particular  activity 
will  begin  and/or  be  completed  is  not 
dependent  upon  (affected  by  or  directed 
by)  Input  (present  or  future)  from  others 
COST  AUTHORITY 

The  degree  to  which  one's  decision  on 
allocations  of  resources  to  a  particular 
activity  is  not  dependent  upon  (affected 
by  or  directed  by)  Input  (present  or 
future)  from  others. 

It  should  be  noted  that  these  dimensions  which 
arise  from  the  boundary  between  "outside"  the 
program  and  "inside"  can  be  replicated  for  any 
similar  boundary,  e.g.,  between  the  program 
manager  and  managers  of  sub-programs. 


E.  COMMENTS 


These  three  categories  of  dimensions  are  pro¬ 
posed  as  a  useful  framework  for  identifying 
the  uncertainties  the  program  manager  faces. 
They  draw  upon,  and  parallel,  categories  which 
have  been  developed  and  tested  and  appear  in 
conventional  wisdom  and  practice.  The  thrust 
of  the  presentation  here  is  to  present  them  In 
terms  of  the  relation  with  uncertainty. 


SECTION  V 


SUMMARY 


This  paper  Is,  and  was  Intended  to  be,  a  dis¬ 
cussion  paper  -  to  present  some  concepts  on 

the  relation  of  uncertainty  to  program  manage¬ 
ment.  The  central  proposal  is  the  use  of  or- 
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dinal  dimensions  to  identify  the  key  parts  of 
uncertainty  associated  with  the  program  plan, 
the  program  itself,  and  the  relationship  of 
the  program  manager  with  others.  This  ap¬ 
proach  is  proposed  to  appropriately  reflect 
not  only  the  nature  of  uncertainty  but  also 
the  salient,  related  characteristics  of  the 
systems  acquisition  process. 


APPENDIX  I 


NOTATIONS 


A .  Presentation  of  "dimensions" 

1.  Where  only  the  name  of  a  dimension  appears 
in  the  text,  reference  may  be  made  to  Appen¬ 
dix  II  for  the  description  of  the  dimension. 

2.  The  names  of  dimensions  appear  in  two 

forms:  some  describe  the  end  point  of  the  di¬ 
mension,  e.g.,  ML  -  R,"  and  others  describe 

one  or  the  other  end  point,  e.g.,  "R." 

3.  The  (word)  description  of  all  dimensions 
is  always  in  terms  of  the  ordinal  dimension, 

i.e.,  "...the  degree  to  which...";  however, 
only  a  few  of  the  dimensions  will  include  un¬ 
certainty. 

4.  The  end  points  of  the  scale  used  for  all 

dimensions  are  described  as  follows:  "Not  at 
all  -  Completely." 

B .  Basic  notations 

1.  X.  The  set  of  all  "things"  can  be  (will 
be)  described  in  terms  of  "any  conceptually 
held  change  in  state."  For  convenience 
"change  in  state"  and  "state"  may  be  consid¬ 
ered  interchangeable,  depending  only  upon  the 
level  of  analysis.  The  notation  will  be  X, 
sometimes  further  described  by  a  subscript, 
e.g.,  X^^. . 

2.  X— X.  The  relationship  between  two  (or 

more)  states  can  be  (will  be)  described  with  a 
horizontal  connecting  line  ( - ),  e.g., 

Xtv- - Xdv.  The  set  of  possible  relationships 

is  not  limited. 

C.  Basic  concepts 

1.  Time.  Changes  in  state  (or  states)  of 
time  are  conventionally  described  as  a  "point 
in  time"  (or  "event")  or  a  "period." 

2.  Space.  Changes  in  state  (or  states)  of 
space  are  conventionally  described  as  a  "point 
in  space"  or  "object." 

3.  Confidence  (uncertainty).  Changes  in 
state  (or  states)  of  confidence  are  conven¬ 
tionally  described  as  "level  of  confidence 
(or  uncertainty)"  or  "increase/decrease  in 
confidence  (or  uncertainty)." 

4.  (Utility).  Changes  in  state  (or  states) 
of  utility  are  conventionally  described  as 
"level  of  preference"  or  "increase/decrease  in 
preference ." 

D.  Other  notations 


1.  DM  (and  PM).  For  convenience  here,  man  is 
considered  a  decision  maker  (DM)  who  not  only 
"holds"  (or  observes)  the  above  changes  in 
state  (or  states)  but  also  may  make  changes 
(decisions)  in  (or  with  respect  to)  those 
states.  Further,  the  term  program  manager 
(PM)  will  be  used  for  the  decision  maker  who 
(primarily)  makes  decisions  about  some  "pro¬ 
gram." 

2.  Program.  The  term  PROGRAM  will  be  used 
for  a  set  of  related  changes  which  occur  (or 
can  occur)  over  a  period  of  time. 

3.  (Time  notations).  With  relation  to  a  pro¬ 
gram,  the  beginning  or  ^initial  point  will  be 
noted  as  t^,  and  the  ending  or  end  point  will 
be  noted  as  te.  In  addition,  times  within  the 
program  will  be  noted  as  tQ,  i.e.,  t^<  tQ<  te; 
and  times  within  the  program  but  including  the 
end  points  will  be  noted  as  tm,  i.e., 

ci  ^  te* 

4.  (Variables  notations).  For  describing  or 
examining  any  change  (in  part  or  all  of  the 
program)  which  is  of  interest,  the  following 
notations  may  be  used: 

Xiv  =  independent  variable  (s) 

Xdv  =  dependent  variable(s) 

xpar  *  parameters) 

Xbkgd  “  background 

Xrc  *  resources  and/or  £onstraints 

xse  =  .subsequent  effects 

Xso  38  £p in-offs. 

APPENDIX  II 


SOME  SELECTED  DIMENSIONS 


UNCERTAINTY  -  CERTAINTY 

The  degree  to  which  a  specified  relationship 
(including  phenomena,  abstract  constructs,  ob¬ 
jects,  events,  etc.)  is  certain. 

LEVEL  OF  CONFIDENCE 

The  degree  to  which  one  is  certain  that  a  par¬ 
ticular  relationship  is  as  stated. 

CONFIDENCE  IN  C 

The  degree  to  which  one  is  certain  that  the 
(perceived)  uncertainty  in  some  specified  re¬ 
lationship  is  as  stated. 

EFFECT  UPON  UNCERTAINTY  (CONFIDENCE) 

The  degree  to  which  the  effect  of  the  rela¬ 
tionship  of  other  variables  (X  )  to  the  re¬ 
lationship  of  the  variables  of  interest 
Xiv»  Xdv^  can  be  (°r  is)  determined. 

PARAMETERS 

The  degree  to  which  a  specified  relationship 
is  not  affected  by  other  relationships. 

LENGTH  OR  DEPTH  (PARAMETERS) 

The  degree  to  which  the  relationship  between 
two  variables  of  interest  (Xiv  and  Xdv)  is 
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limited  to  the  "same"  time,  i.e.,  not  separ¬ 
ated  in  time  to  the  extent  that  other  vari¬ 
ables  (Xpar)  may  "affect"  the  relationship. 

BREADTH  (PARAMETERS) 

The  degree  to  which  the  relationship  between 
a  variable  of  interest  (Xd  )  and  the  varia¬ 
ble  (s)  (X. v,  X  )  which  may  "affect"  it  can 
be  (or  is)  limited  to  a  single  relationship 
at  the  "same"  time. 

UNSTRUCTURED  -  STRUCTURED 

The  degree  to  which  the  variables  of  interest 
(Xiv  and  Xdv),  other  variables  (Xpar)  which 
may  "affect"  their  relationship,  and  the  re¬ 
lationships  among  them  can  be  identified  (or 
described,  or  are  certain). 

OPEN  ----  CLOSED 

The  degree  to  which  the  variables  of  interest 
(Xiv,  Xdv)  and  other  variables  (X  ar)  which 
may  "affect"  their  relationship  can  be  dis¬ 
tinguished  from  the  set  of  other  variables 
(Xbkgd)- 

KNOW  ABLE 

The  degree  to  which  the  specified  relationship 
is  capable  (at  some  specified  level  of  cost 
and  time)  of  being  certain. 

RETROSPECTIVE  ----  PROSPECTIVE 
The  degree  to  which  the  phenomenon  of  interest 
occurs  much  later  rather  than  much  earlier 
than  the  time  of  some  decision  (or  observa¬ 
tion)  . 

EXPLORATORY  ----A  PRIORI 

The  degree  to  which  the  researcher  predeter¬ 
mines  the  effect  of  new  data  on  his  results. 

STUDY  ----  EXPERIMENT 

The  degree  to  which  the  researcher  manipulates 
the  phenomena  under  observation. 

NORMATIVE  — —  EMPIRICAL 

The  degree  to  which  the  researcher  obtains 
new  data  directly  from  the  phenomena. 

FACT  — --  VALUE 

The  degree  to  which  uncertainty  with  respect 
to  a  specific  relationship  is  not  shared 
across  a  set  of  individuals. 

SUBJECTIVE  ----  OBJECTIVE 

The  degree  to  which  the  source  of  data  is  not 
the  assertion  of  a  specific  individual. 

NOMINAL  — -  RATIO 

The  degree  to  which  the  set  of  data  is  inter¬ 
related  (and  can  be  manipulated  mathemati¬ 
cally)  . 

VALIDATION  -  MULTI-METHOD 

The  degree  to  which  the  relationship  is  de¬ 
termined  by  the  convergence  of  measures  ob¬ 
tained  by  "many"  methods. 


VALIDATION  -  MULTI-TRAIT 

The  degree  to  which  the  relationship  is  de¬ 
termined  by  the  convergence  of  measures  of 
"many”  traits. 

LEVEL  OF  CONFIDENCE  (WITH  RESPECT  TO  UTILITY) 
The  degree  to  which  one  is  certain  of  the 
relationship  between  a  specified  state  (ex¬ 
isting,  planned,  or  expected)  and  one's  pre¬ 
ferred  state. 

EFFECT  UPON  UTILITY  (PURPOSE,  GOALS,  ETC.) 

The  degree  to  which  the  variables  of  interest 
(Xiv»  Xdv)  are  related  to  other  variables 
which  are  related  to  utility. 

RESOURCES/ CONSTRAINTS 

The  degree  to  which  a  variable  or  interest 
(X^v)  ^-s  related  to  (dependent  upon)  some  other 
variables  which  are  related  to  utility. 

SUBSEQUENT  EFFECT 

The  degree  to  which  a  variable  of  interest 
(Xdv)  is  related  to  some  other  variable  (Xge) 
which  is  related  to  utility. 

SPINOFF 

The  degree  to  which  some  other  variables  (Xso) 
which  are  related  to  utility  are  related  (de¬ 
pendent  upon)  the  variables  of  interest 

<Xiv’  Xdv>* 

LEVEL  OF  CONFIDENCE  (DECISION  MAKING) 

The  degree  to  which  the  (perceived)  uncer¬ 
tainty  with  respect  to  some  relationship  is 
equal  to  (or  less  than)  that  required  for  the 
purposes  of  some  decision. 

LEVEL  OF  CONFIDENCE  WITH  RESPECT  TO  UTILITY 
(DECISION  MAKING) 

The  degree  to  which  the  (perceived)  uncer¬ 
tainty  with  respect  to  the  relationship  with 
one's  preferred  state  is  equal  to  (or  less 
than)  that  required  for  the  purposes  of  some 
decision. 

AUTHORITY/ INDEPENDENCE  (WITH  RESPECT  TO  UN¬ 
CERTAINTY) 

The  degree  to  which  the  identification  of 
other  variables  (X  ar)  which  may  be  related 
(affect)  the  relatfonship  of  interest 
(Xfv  —  Xdv),  and  the  determination  of  their 
relationship,  is  not  dependent  upon  (affected 
by  or  directed  by)  input  (present  or  future) 
from  others. 

AUTHORITY/ INDEPENDENCE  (WITH  RESPECT  TO 
UTILITY) 

The  degree  to  which  the  identification  of  the 
variables  which  are  related  to  one's  preferred 
state,  and  the  determination  of  their  rela¬ 
tionship,  is  not  dependent  upon  (affected  by 
or  directed  by)  input  (present  or  future)  from 
others . 
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KNOWLEDGE,  SKILLS  AND  PROBLEM  SOLVING  ABILITY 
The  degree  to  which  the  decision-maker's 
awareness  and  understanding  of  the  relevant 
states  of  nature  and/or  prior  art  is  equal  to 
(or  exceeds)  that  required  for  the  purpose  of 
some  decision. 

TOLERANCE  OF  AMBIQUITY 

The  degree  to  which  the  uncertainty  in  a  spec¬ 
ified  (or  set  of  specified)  relationship (s)  is 
equal  to  (or  less  than)  that  required  for  the 
purposes  of  some  decision. 

NEED  TO  STRUCTURE 

The  degree  to  which  the  uncertainty  in  a  spec¬ 
ified  (or  set  of  specified)  relationship (s)  is 
equal  to  (or  greater  than)  that  required  for 
the  purposes  of  some  decision. 

RISK  AVERSION 

The  degree  to  which  the  uncertainty  in  a  spec¬ 
ified  relationship  is  equal  to  (or  greater 
than)  that  required  for  the  purposes  of  some 
decision. 

FEASIBILITY  (CREDIBILITY) 

The  degree  to  which  the  (perceived)  uncertain¬ 
ty  with  respect  to  some  relationship  is  equal 
to  (or  more  than)  that  required  for  some  de¬ 
cision. 

TIMELINESS 

The  degree  to  which  the  (perceived)  uncer¬ 
tainty  with  respect  to  some  relationship  is 
equal  to  (or  less  than)  that  required  for  some 
decision  at  (or  before)  some  specified  time  of 
decision. 

COSTLINESS 

The  degree  to  which  the  (perceived)  uncertain¬ 
ty  with  respect  to  some  relationship  is  equal 
to  (or  less  than)  that  required  for  some  de¬ 
cision  at  (or  less  than)  some  specified  level 
of  cost. 
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RISK  AND  UNCERTAINTY  IN  NETWORK  ANALYSIS  ACTIVITY  ESTIMATES 

Dr.  John  R.  Adams,  Western  Carolina  University 
Ms.  Jo  Karen  Busch,  Western  Carolina  University 


ABSTRACT 


Previous  research  on  the  accuracy  of  activity 
time  estimates  in  programs  using  network  based 
management  systems  are  reviewed,  and  the  con¬ 
flicting  results  of  previous  research  are  docu¬ 
mented.  Two  studies  designed  to  extend  the  pro¬ 
gram  manager's  knowledge  of  factors  affecting 
estimating  accuracy,  and  to  integrate  previous 
findings,  are  summarized.  The  earlier  conflict¬ 
ing  results  are  explained,  and  managerial  ac¬ 
tions  necessary  to  reduce  the  level  of  uncer¬ 
tainty  inherent  in  network  estimates  are  sug¬ 
gested  , 


INTRODUCTION 


The  use  of  network  analysis  techniques  on  mili¬ 
tary  programs  and  projects  began  with  applying 
the  Program  Evaluation  and  Review  Technique 
(PERT)  to  the  Navy's  Polaris  Porgram  in  the  ear¬ 
ly  1960s.  Since  that  time  network  analysis  has 
developed  through  a  number  of  modifications. 

The  DOD’s  current  Cost/Schedule  Control  System 
Criteria  (C/SCSC)  allows  flexibility  In  select¬ 
ing  the  specific  technique  to  be  used.  Since 
the  days  of  Polaris,  however,  the  prime  con¬ 
tractor  and  major  subcontractors  on  almost 
every  major  weapon  system  development  program 
have  been  required  to  use  some  form  of  network- 
based  planning  and  progress  reporting  technique. 
The  purpose  has  been  to  reduce  the  uncertainties 
involved  in  planning  and  executing  major  weapon 
system  programs.  The  continuing  cost  and  sched¬ 
ule  overruns  experienced  by  major  programs  dem¬ 
onstrates  that  at  least  some  sources  of  uncer¬ 
tainty  are  being  inadequately  dealt  with. 

In  applying  network  analysis,  the  total  program 
is  divided  into  relatively  small  and  independent 
activities.  The  duration  and  resources  required 
for  each  activity  are  estimated.  These  activi¬ 
ties  are  then  assembled  into  a  time-phased  net¬ 
work  or  "map"  for  analysis  of  the  overall  pro¬ 
gram.  Thus  the  purpose  of  using  network  anal¬ 
ysis  techniques  lies  in  reducing  program  uncer¬ 
tainties— improving  the  accuracy  with  which  the 
overall  program  schedule  and  resource  require¬ 
ments  can  be  estimated  and  managed.  These  un¬ 
certainties  can  obviously  be  reduced  only  to  the 
extent  that  the  underlying  activity  estimates 
are  accurate.  Much  has  been  written  about  the 
ability  to  develop  and  meet  program  schedules 
using  network  techniques,  yet  only  a  very  lim¬ 
ited  literature  exists  dealing  with  the  accuracy 
of  individual  activity  estimates  and  the  effect 
of  uncertainties  in  these  estimates  on  the  over¬ 
all  program. 


A  time  estimate  has  been  described  as  the  man¬ 
ager's  experience  and  knowledge  reflected  in  a 
numerical  guess  (4,  p.  87).  If  this  view  is 
accepted,  then  uncertainties  in  the  estimates 
must  be  influenced  by  characteristics  of  both 
the  activity  itself  and  the  estimator.  The 
studies  summarized  in  this  paper  examine  these 
characteristics  to  determine  the  effect  they 
have  on  activity  estimates  and  on  the  level  of 
uncertainty  involved  in  the  overall  project  or 
program.  The  purpose  is  to  provide  the  back¬ 
ground  necessary  to  develop  a  classification 
scheme  for  projects  which  will  allow  the  man¬ 
ager  to  roughly  forecast  the  uncertainty  inher¬ 
ent  in  his  project  estimates. 


BACKGROUND 


Time  estimates  for  individual  project  activi¬ 
ties  are  critical  to  the  use  of  network  analysis 
techniques.  They  form  the  basis  of  all  analyt¬ 
ical  operations  carried  out  on  the  network,  and 
the  resulting  work  schedule  can  be  no  more  ac¬ 
curate  than  the  basic  estimates  from  which  it 
is  calculated.  It  is  therefore  highly  advanta¬ 
geous  to  obtain  the  most  accurate  activity  es¬ 
timates  feasible.  The  difficulty  of  developing 
accurate  estimates  is  inherent  in  the  high  tech- 
nology,  advanced  development  type  of  project 
characteristic  of  weapon  system  development 
programs,  and  for  which  network  analysis  tech¬ 
niques  are  most  suited.  Absolute  accuracy  is 
virtually  impossible  to  achieve  (17,  p.  87) 
since  the  previous  experience  on  which  estimates 
would  normally  be  based  either  does  not  exist 
or  does  not  exactly  apply.  It  is  inevitable, 
therefore,  that  individual  time  estimates  will 
contain  errors. 

Current  research  directed  at  investigating  the 
accuracy  of  activity  estimates  for  project  net¬ 
works  centers  on  two  main  sources  of  error;  Che 
quantitative  assumptions  of  the  network,  and  the 
estimating  behavior  of  those  managers  who  must 
provide  the  estimates.  In  the  quantitative 
area,  MacCrimmon  and  Ryavec  (14)  published  what 
is  considered  the  classic  survey  of  error 
sources  in  the  network  assumptions.  Other  au¬ 
thors  have  recommended  changes  to  the  basic 
network  calculations  in  order  to  reduce  these 
errors  (11;  13).  These  changes  have  not  a- 
chieved  wide  acceptance,  possible  because  of 
the  complexity  they  introduce  into  the  tech¬ 
niques.  While  the  basic  quantitative  assump¬ 
tions  do  lead  to  significant  errors  in  the  over¬ 
all  network  predictions  (14) ,  they  can  be  com¬ 
pensated  for  by  using  a  Monte  Carlo  sampling 
approach  to  predicting  the  project  durations 
(5).  However,  they  have  little  effect  on  the 
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individual  activity  estimates  and  the  uncertain¬ 
ty  created  for  the  project  by  activity  estima¬ 
tion  errors. 

Research  on  the  estimating  behavior  of  managers 
has  principally  investigated  the  existence  of  a 
learning  curve  to  determine  if  estimates  improve 
as  the  project  progresses,  as  the  estimator's 
experience  with  the  project  increases,  or  as  the 
activity  start  date  approaches.  The  existence 
of  a  consistent  learning  curve,  or  indeed  any 
consistent  bias,  might  allow  the  estimates  to 
be  adjusted  based  on  a  history  of  the  estima¬ 
tor's  performance  and  on  the  activity's  rela¬ 
tive  location  during  the  project.  Archibald 
and  Villoria  (4)  report  that  the  Navy  conducted 
extensive  studies  of  activity  time  estimates  in 
the  early  1960s  as  part  of  the  Polaris  Missle 
Project.  They  compared  the  actual  with  the  es¬ 
timated  activity  times  and  found  that  the  esti¬ 
mates  tended  to  fall  short  of  the  time  required 
to  complete  the  activity.  As  the  estimators 
gained  experience  over  a  two-year  period,  how¬ 
ever,  activity  time  estimates  became  increas¬ 
ingly  realistic,  implying  the  existence  of  the 
learning  curve  effect.  More  recently,  Abernathy 
(1),  using  data  obtained  from  a  f irst-of-a-kind 
electronic  space  module  project,  found  estima¬ 
ting  accuracy  was  a  function  of  several  vari¬ 
ables  taken  together:  the  estimate's  nearness 
in  time  to  the  activity  start  rate,  the  esti¬ 
mate's  nearness  in  time  to  the  project  start 
date,  the  number  of  completed  project  activi¬ 
ties,  and  the  error  in  previous  estimates  of  the 
same  activity's  duration.  Of  course,  this  last 
variable  cannot  be  known  until  after  the  activ¬ 
ity  has  been  completed,  a  fact  which  would  limit 
the  application  of  his  findings.  Abernathy  used 
the  number  of  days  difference  between  the  esti¬ 
mated  and  actual  activity  durations  as  his  mea¬ 
sure  of  "estimation  error". 

On  the  other  hand,  King  and  his  various  col¬ 
leagues  (10;  11;  12)  report  that  no  such  learn¬ 
ing  curve  exists.  Based  on  data  obtained  from 
a  U.S.  Air  Force  weapon  system  acquisition  pro¬ 
ject,  King  and  Wilson  (12)  concluded  that  no 
improvement  in  the  accuracy  of  time  estimates 
exists  as  the  activity  start  or  completion  date 
approaches.  Kidd  and  Morgan  (9)  supported  this 
conclusion  in  their  study  of  overhauls,  con¬ 
struction,  and  plant  reliability  work  controlled 
by  the  Midlands  Regions  of  the  Central  Electric¬ 
ity  Generating  Board  in  Great  Britain.  The  con¬ 
clusions  reached  in  the  above  studies  were  all 
based  on  using  the  ratio  of  estimated  to  actual 
times  as  the  measure  of  accuracy. 

There  has,  of  course,  been  much  generalized  re¬ 
search  conducted  on  the  motivation  of  profes¬ 
sionals  and  managers.  For  example,  both  Mc¬ 
Clelland  (15)  and  Kahn  (8)  have  reported  that 
managers  and  business  executives  appear  partic¬ 
ularly  driven  by  a  high  need  for  achievement. 
Meyer's  work  (16)  indicates  that  managers  im¬ 
prove  their  performance  most  when  specific 
goals  are  established.  Hall  and  Lawler  (7)  have 


demonstrated  that  pressures  for  quality  (a  pro¬ 
fessional  concern)  and  financial  responsibility 
(an  organizational  concern)  are  both  related  to 
the  organizational  performance  of  professionals. 
No  research  was  found  which  specifically  relat¬ 
ed  individual  estimator  characteristics  to  the 
accuracy  of  the  activity  estimates  they  pro¬ 
duced. 

Beyond  the  above  referenced  research,  little 
has  been  done  to  investigate  what  factors  af¬ 
fect  the  accuracy  of  activity  time  estimates. 
Since  the  data  for  studying  the  learning  curve 
phenomena  were  drawn  from  widely  different  pro¬ 
jects,  and  a  variety  of  analysis  techniques 
were  used  in  the  studies,  there  seemed  to  be  no 
way  to  reconcile  the  conflicting  results  with¬ 
out  additional  research.  Only  by  reconciling 
these  differences  could  generalizab le  results 
be  obtained  which  would  aid  the  practicing  pro¬ 
ject  manager  to  obtain  more  accurate  activity 
estimates.  Further,  this  line  of  research 
seemed  sterile  in  its  failure  to  consider  the 
impact  of  individual  differences  among  esti¬ 
mators  . 


THE  STUDIES 


Two  studies  were^ccmduc ted  to  deal  with  the 
concerns  identified  above.  These  studies  have 
been  reported  elsewhere  separately  and  in  de¬ 
tail  (3;  2)  so  are  only  summarized  below. 
First,  however,  it  is  important  to  understand 
three  separate  aspects  of  the  estimating 
situation. 

Project  Differences 

All  of  the  previously  reported  studies  drew 
their  conclusions  from  analyses  of  single  pro¬ 
jects,  yet  projects  by  their  nature  differ 
markedly  in  terms  of  the  uncertainty  involved 
in  the  technical  aspects  of  the  work  being 
performed.  The  level  of  technical  uncertainty 
in  a  maintenance  or  construction  project,  for 
example,  would  generally  be  accepted  as  lower 
than  that  in  an  advanced  research  and  develop¬ 
ment  project.  If  generalizable  results  are  to 
be  obtained,  such  critical  cross-project 
variables  must  be  addressed  even  if  this  can 
only  be  accomplished  subjectively. 

Time  Relationships 

There  is  really  no  single  time  estimate  for  an 
activity.  Rather,  a  series  of  estimates  are 
made  for  each  activity.  Most  projects  require 
an  updated  estimate  each  two  to  four  weeks. 
Note,  however,  that  each  estimating  situation 
is  different  (see  Figure  1).  As  the  project 
proceeds,  the  estimate  is  projected  over  a 
shorter  period  of  time  (imminence);  there  is 
more  experience  with  and  knowledge  of  the  pro¬ 
ject  (project  progress);  there  is  also  a  con¬ 
tinual  increase  in  knowledge  about  the  techni- 
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cal  requirements  of  the  activity  and  about  pro¬ 
gress  on  related  activities.  After  the  start 
of  the  activity  (activity  start  date) ,  the  task 
being  estimated  changes  as  portions  of  it  are 
completed,  while  knowledge  of  progress  on  and 
problems  with  the  activity  become  available. 

Thus  each  estimate  reflects  a  separate  situation 
and  should  be  considered  separately  in  studies 
of  estimating  behavior.  The  following  indepen¬ 
dent  variables  define  these  time  relationships: 


The  following  three  measures  of  accuracy  were 
used  to  allow  interpretation  of  both  the  magni¬ 
tude  and  direction  of  estimating  errors,  and  a 
comparison  of  the  measures  used  by  both  King  and 
Abernathy: 

1.  Accuracy  # 1  is  the  time  by  which  the 
actual  time  (A  ,  )  exceeded  or  was  less  than  the 
activity  time  estimate  (E. .,  )  for  estimate  i  of 
activity  j  in  project  k,  and  provides  the  error 
in  terms  of  days. 


FIGURE  1 
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TIME  RELATIONSHIPS  IN  THE  ESTIMATING  SITUATION 


Project  Estimated 


1. 


Imminence  (I  ), 

xj  is 


Imminence  is  the 


nearness  in  time  of  the  estimate  date  (ED)  to 
the  activity  start  date  (ASD)  for  estimate  i 
of  activity  j  in  project  k. 


V  (ASD)jk  -  (ED)ijk *  (1) 

2.  Project  Progress  (P^)  ■  ProJect  pro_ 

gress  represents  "experience"  with  the  project. 
It  is  the  difference  between  the  project  start 
date  (PSD)  and  the  estimate  date  (ED)  for  esti¬ 
mate  i  of  activity  j  in  project  k. 


2.  Accuracy  #2  is  the  absolute  value  of 
the  difference  between  the  activity  time  esti¬ 
mate  (E,  ,  )  and  the  activity  time  (A..  )  for  es¬ 
timate  iJof  activity  j  in  project  k.  This  pro¬ 
vides  the  error  in  days  ignoring  the  difference 
between  positive  and  negative  errors. 


(ACC  *2).Jk  = 


jk 


-  E 


ijk 


(4) 


3.  Accuracy  #3  is  the  ratio  of  the  activi¬ 
ty  time  estimate  (E  .)  to  the  actual  activity 
time  (A.,)  for  estimate  i  of  activity  j  in  pro¬ 
ject  k.J  ihis  provides  a  measure  of  accuracy  as 
a  ratio,  with  perfect  accuracy  indicated  by  a 
value  of  one,  optimistic  estimates  (the  result 
of  underestimation)  indicated  by  values  from 
zero  to  one,  and  pessimistic  estimates  (the  re¬ 
sult  of  overestimation)  indicated  by  values 
greater  than  one.  This  measure  of  accuracy  is 
clearly  nonlinear  with  respect  to  time. 


(ACC  #3) 


ijk 


E,  .i/A,, 

ijk  jk 


(5) 


ACC  //I  was  previously  used  in  the  Abernathy 
study,  while  ACC  #3  was  used  by  King  and  his 
associates . 


pijk  =  (ED)ijk  -  (PSD)k  (2) 


3.  Activity  Time  (A.,  ).  Activity  time 
J  K 

represents  the  actual  duration  of  activity  j 
in  project  k. 


4. 


Estimated  Activity  Time 


(Enk>' 


Estimated  activity  time  is  the  duration  fore¬ 
casted  by  estimate  i  for  activity  j  in  project 


k. 


Accuracy  Measures 

The  dependent  variable  of  interest  represents 
the  deviation  of  the  estimate  from  the  actual 
activity  time.  Accuracy  has  several  aspects, 
any  one  of  which  may  be  critical  at  different 
times.  The  absolute  value  of  the  error  may  be 
useful  for  evaluating  an  estimator’s  ability 
and  assigning  the  "best"  estimator  to  the  most 
critical  activities,  for  example.  Further,  the 
amount  and  direction  of  error  may  be  critical 
to  scheduling  and  allocating  scarce  resourses. 


Study  I 

The  first  study  was  designed  to  investigate  the 
conflicting  results  reported  by  King  and  Aber¬ 
nathy.  Data  were  obtained  on  over  100  activi¬ 
ties  from  each  of  three  separate  and  diverse  Air 
Force  sponsored  projects  carefully  selected  to 
represent  a  wide  range  of  technical  uncertainty. 
For  each  project,  all  available  estimates  for 
all  completed  activities  were  included  in  the 
sample.  This  provided  a  total  of  936  estimates 
for  381  separate  activities.  The  projects  se¬ 
lected  for  study  are  typical  of  many  large, 
government-sponsored  efforts  conducted  under 
contract  by  several  private  corporations  and 
with  very  close  government  supervision.  Project 

1  involved  a  major,  advanced  "state-of-the-art" 
modification  to  an  existing  weapon  system,  and 

clearly  represented  a  relatively  high  level  of 
technical  uncertainty.  Project  2  involved  the 
design  and  implementation  of  an  advanced  manage¬ 
ment  information  system,  probably  reflecting  an 
intermediate  level  of  technical  uncertainty. 
Project  3  involved  the  design  and  construction 
of  an  advanced  engineering  test  facility  which 
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was  unusual  in  design  but  clearly  within  the 
"state-of-the-art".  This  project  represented  a 
relatively  moderate  level  of  uncertainty. 

The  analysis  began  by  grouping  the  initial  esti¬ 
mates  for  each  activity  according  to  their  opti¬ 
mistic  or  pessimistic  bias.  Table  1  demon¬ 
strates  that  the  majority  of  the  activities  for 
all  projects  were  underestimated  with  relatively 
few  compensating  overestimations,  providing  sup¬ 
port  for  the  frequently  reported  optimistic  bias 
in  network  estimates.  The  inference  can  be 
drawn  that  if  no  learning  exists,  all  projects 
would  be  expected  to  exceed  their  estimated 
completion  dates. 

TABLE  1 

PERCENTAGE  OF  ACTIVITIES  BY  PROJECT  IN  WHICH 
THE  initial  ESTIMATES  UNDERESTIMATED, 
OVERESTIMATED,  OR  CORRECTLY  ESTIMATED 
THE  ACTUAL  ACTIVITY  DURATION 


PROJECT 

1  2  3 

COMBINED 

DATA 

UNDERESTIMATED 

(OPTIMISTIC) 

52.5  52.4  44.1 

49.7 

OVERESTIMATED 

(PESSIMISTIC) 

27.7  3.9  15.7 

15.7 

CORRECT 
(±  ONE  DAY) 

19.3  43.7  40.2 

34.6 

TABLE  2 


ACCURACY  OF  THE  ACTIVITY  TIME  ESTIMATES 
BY  PROJECT  AND  MEASURE  OF  ACCURACY* 


acc  n 

ACC  1t2 

ACC  #3 

MOST  ACCURATE 

Proj.  1 

Proj.  2 

Proj.  3 

Proj.  3 

Proj.  1 

Proj.  2 

LEAST  ACCURATE 

Proj.  2 

Proj.  3 

Proj.  1 

Rank  ordered  based  on  the  mean  values 
achieved  by  the  respective  accuracy  measures 
for  the  individual  projects. 

Table  1  also  implies  that  Project  1  was  the 
least  accurately  estimated.  Table  2,  however, 
demonstrates  that  any  of  the  three  projects  can 
be  the  "most  accurately  estimated",  depending 
on  the  measure  of  accuracy  used.  Such  an  out¬ 


come  should  not  be  surprising.  ACC  #1  measures 
the  actual  number  of  days  error  in  the  estimate. 
Thus  a  ten-week  and  a  one -week  activity,  both 
underestimated  by  one  day,  would  carry  the  same 
weight  in  determining  the  mean  value  of  all 
estimates.  The  direction  of  the  error  also 
influences  the  value  of  ACC  #1.  For  example, 
a  one-day  overestimate  and  a  one-day  underesti¬ 
mate  cancel  each  other  out,  resulting  in  a 
mean  error  of  zero.  In  ACC  //3,  however,  the 
ratio  E/A  allows  the  duration  of  the  activity 
to  influence  the  results.  A  one-day  error 
would  be  more  serious  in  a  one-week  than  in  a 
ten-week  activity.  ACC  #2  produces  a  mean  of 
the  absolute  values  of  the  deviations  of  actual 
from  estimated  times,  eliminating  consideration 
of  the  direction  of  the  error.  The  conflicting 
results  shown  in  Table  2  serve  to  point  out 
that  it  is  possible  for  both  the  King  and 
Abernathy  findings  to  be  correct  for  the 
measures  of  accuracy  each  used.  While  the 
critical  path  could  not  be  taken  into  account 
in  these  calculations,  the  practical  management 
of  projects  and  programs  requires  resources  be 
allocated  to  each  activity  being  conducted. 

The  accuracy  of  each  estimate  is  therefore  im¬ 
portant  to  the  project.  Further,  since  these 
resources  must  be  allocated  on  a  calendar  day 
basis,  ACC  #1  would  appear  to  be  the  most  use¬ 
ful  indicator  of  accuracy  to  the  practitioner. 

The  correlations  among  the  accuracy  measures 
and  imminence,  activity  time,  and  project  pro¬ 
gress  are  presented  in  Table  3.  The  findings 
are  supported  by  regression  analyses.  The  only 
independent  variable  that  was  consistently  sig¬ 
nificant  across  projects  was  activity  time. 
Scattergrams  further  indicate  that  the  esti¬ 
mates  for  long  duration  activities  tend  to  be 
less  accurate  than  those  for  shorter  activi¬ 
ties.  The  nonsignificant  and  relatively  low 
correlations  between  ACC  #3  and  other  variables 
are  explained  by  the  fact  that  correlation  is 
a  measure  of  the  linear  association  between  two 
variables.  Since  ACC  ,f3  is  non-linear,  high 
correlation  coefficients  between  it  and  other 
variables  are  not  to  be  expected. 

Abernathy,  in  his  study  using  ACC  it  1,  found  a 
pronounced  learning  curve.  In  this  study,  the 
high  correlations  between  accuracy  and  imminence 
tend  to  confirm  his  finding,  but  only  for  Pro¬ 
ject  1.  This  is  the  project  that  most  nearly 
duplicates  the  type  of  project  Abernathy  used  as 
a  data  source.  Both  projects  apparently  demon¬ 
strated  a  high  level  of  technical  uncertainty. 
King  et  al.,  in  their  studies  using  ACC  // 3  and 
the  independent  variable  imminence,  failed  to 
find  a  learning  curve.  This  study  confirms 
their  findings  as  well,  but  only  for  Projects  2 
and  3.  King  et  al.  do  not  provide  sufficient 
information  about  their  data  source  to  evaluate 
the  level  of  technical  uncertainty,  but  Projects 
2  and  3  in  this  study  certainly  involve  signi¬ 
ficantly  less  technical  uncertainty  than  does 
Project  1.  The  existence  of  a  learning  curve  is 
clearly  nroiect  dependent.  The  data  presented 
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TABLE  3 


SUMMARY  OF  RELATIONSHIPS  IDENTIFIED 
FROM  CORRELATIONS  BY  PROJECT 
AND  ACCURACY  MEASURES* 


Accu- 

racy 

Measure 

Number 

Significant  Relationships 

Imminence 

Activity 

Time 

Proj  ect 
Progress 

Proj  . 

1 

- . 31  (.99) 

. 50 ( . 99) 

NS 

No.  1 

2 

. 18 ( . 95) 

NS 

-. 24( . 98) 

3 

.24(98) 

-. 28 (. 99) ** 

NS 

Proj . 

1  . 

NS 

. 9  9 ( . 99) 

NS 

No.  2 

2 

NS 

. 99 ( . 99) 

— . 20 ( . 95) 

3 

NS 

NS** 

NS 

Proj. 

1 

NS 

. 56 ( . 99) 

NS 

No.  3 

2 

NS 

. 61 (. 99) 

NS 

3 

NS 

-. 20(.95)** 

NS 

Com¬ 

1 

NS 

. 51 (. 99) 

NS 

bined 

2 

NS 

. 51 ( . 99) 

NS 

Data 

3 

NS 

-. 18 ( . 95)** 

Simple  Pearson  Product  Moment  correlations. 
The  statistical  significance  (1  -«C)  ±s  indi¬ 
cated  in  parentheses  for  each  significant 
relation.  NS  indicates  significance  (1  -  £-) 
below  0.95. 

** 

Regression  analysis  indicates  a  highly  sig¬ 
nificant  nonlinear  relationship  between  accu¬ 
racy  measure  // 3  and  activity  time  for  all 
projects. 


here  strongly  implies  that  the  existence  of  a 
learning  curve  may  be  related  to  the  level  of 
technical  uncertainty  inherent  in  the  project. 

Study  II 

The  second  study  was  designed  to  investigate  the 
impact  on  estimating  accuracy  of  factors  which 
reflect  individual  estimator  differences  but 
which  might  be  influenced  by  management.  It  was 
hypothesized  that  certain  personality,  situa¬ 
tional,  and  decision  methodology  factors  could 
affect  the  estimator's  accuracy.  A  model  was 
developed  to  graphically  demonstrate  the  hypoth¬ 
esized  relationships  and  the  effect  of  uncon¬ 
trollable  external  factors  on  these  variables. 

In  essence,  this  model  (see  Figure  2)  indicates 
that  managerial  talent,  both  directly  and  in 
combination  with  the  estimator's  perceived  im¬ 
portance  of  accuracy,  influences  the  gathering 
and  processing  of  information  to  affect  esti¬ 
mating  accuracy. 

The  project  selected  for  study  was  again  typical 
of  large,  government-sponsored  efforts,  in  this 
case  involving  three  government  and  six  large 
private  industry  corporations.  Its  purpose  was 
to  supply  advanced  state-of-the-art  aerospace 
technology  to  a  developing  aircraft  system,  a 
task  involving  moderate  levels  of  technical  un¬ 
certainty.  Ninety-four  percent  of  the  individ¬ 
uals  who  had  provided  initial  estimates  of  ac¬ 
tivity  durations  were  studied  in  depth.  The 
project  network  records  yielded  data  for  calcu¬ 
lating  each  estimator's  accuracy  index  as  the 
mean  value  of  his  estimating  errors.  Data  from 
project  records  and  extensive  personal  surveys 
were  analyzed  using  correlation  analysis,  while 
data  from  intensive  interviews  proved  invaluable 
for  interpreting  the  statistical  results. 


FIGURE  2 


MODEL  OF  THE  SIGNIFICANT  RELATIONSHIPS3 


Uncontrollable 

External  Factors 

l 

- 

i 

Data 

i 

Concerning 

1 

The  Activitv 

hypothesized 
relationship 
^  relationships 
assumed  but  not 
testable 


aThe  significance  (1  -«*)  levels  achieved  for  significant  relation¬ 
ships  are  shown  in  parentheses. 
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TABLE  4 


SIMPLE  PEARSON  CORRELATION  COEFFICIENTS  AND  SIGNIFICANCE  (1  -oc) 
LEVELS  BETWEEN  THE  MAJOR  STUDY  VARIABLES 


Perceived 
Importance 
of  Accuracy 

Information 
Sought  and 

P  rucosscd 

Accuracy  Indicc.. 

#] 

if  2 

Managerial  Talent 

0.405 

(.987) 

0.534 

(.997) 

0.383 

(.967) 

0.223 

(.852) 

0.388 
(.9  (;•.’) 

Perceived 

Importance  of 
Accuracy 

— 

0.661 

(.999) 

0.31,8 

(.943) 

0.405 

(.980) 

0.234 

(-875) 

Information 

Sought  and 
Processed 

— 

_ 

0.410 

(.979) 

0.679 

(.999) 

0.551 

(.998) 

Accuracy  Index  \il 

— 

— 

0.406 

(.980) 

— 

— 

Accuracy  Index  #3 

— 

— 

0.  703 
(.999) 

0.642 

(.999) 

— 

"Managerial  talent"  represents  the  sum  of  scores 
for  nine  personality  characteristics,  catego¬ 
rized  as  abilities,  personality  traits,  and 
motivational  traits,  which  have  been  shown  by 
Ghiselli  (6)  to  be  highly  correlated  with  man¬ 
agerial  success.  All  members  of  the  sample  were 
successful  managers  holding  responsible  posi¬ 
tions  and  were  expected  to  obtain  a  certain  min¬ 
imum  score.  Above  this,  however,  extensive 
variation  was  noted  in  the  measure.  "The  per¬ 
ceived  importance  of  accurate  estimates"  repre¬ 
sents  the  extent  to  which  the  estimator  per¬ 
ceives  the  accuracy  of  his  estimates  to  be  im¬ 
portant  to  the  organization,  to  his  career,  and 
to  himself  as  an  individual.  "Information 
sought  and  processed"  represents  a  collection  of 
indicators  which  measure  the  amount  and  type  of 
technical  and  human  information  collected,  the 
methods  employed  in  analyzing  that  information, 
and  the  experience  of  the  estimator. 

The  correlation  matrix  of  results  is  presented 
in  Table  4,  while  the  key  results  are  indicated 
on  the  model  in  Figure  2.  The  relationships  in 
the  model  were  strongly  supported.  The  high 
correlations  among  the  three  accuracy  measures 
supports  the  following  interpretations  of  the 
results . 

Accuracy  is  highly  associated  with  information 
sought  and  processed  for  all  accuracy  measures. 
While  the  more  accurate  estimators  estimate  more 
precisely  (ACC  $2)  ,  they  also  tend  to  underesti¬ 
mate  slightly  (ACC  // 3).  Apparently  those  who 
develop  the  greatest  information  base  relative 
to  the  activity  being  estimated  have  more  con¬ 
fidence  in  their  estimates,  for  they  use  a 
smaller  "contingency  factor"  or  "pad"  (some  said 
they  used  no  contingency  factor  at  all) .  While 


all  respondents  indicated  that  this  relationship 
should  exist,  all  mentioned  some  uncontrollable 
factor  such  as  weather,  strikes,  or  political 
actions,  which  would  cause  the  estimate  not  to 
hold.  Yet  regression  analysis  demonstrates  that 
over  65  percent  of  the  variation  in  accuracy  is 
explained  by  the  information  sought  and  pro¬ 
cessed. 

Information  sought  and  processed  is  also  highly 
associated  with  the  perceived  importance  of  ac¬ 
curacy.  A  more  detailed  analysis  demonstrates 
that  the  principle  components  of  these  two  vari¬ 
ables  are  all  significantly  related  (I  -ot) 
above  the  .95  level,  supporting  the  association 
between  the  primary  variables.  Those  individ¬ 
uals  who  believed  that  accurate  estimates  were 
important  to  themselves,  their  career,  or  their 
organization  devoted  more  time  and  effort  to 
gathering  information  for  their  estimates.  This 
is  an  area  that  could  be  readily  influenced  by 
managerial  personnel.  Yet  not  one  of  the  nine 
organizations  involved  had  any  formal  system  in 
effect  to  provide  information  to  the  estimator 
on  the  accuracy  of  his  estimates,  or  to  differ¬ 
entially  reward  the  accurate  estimator. 

Managerial  talent,  as  defined  by  Ghiselli,  is 
highly  correlated  with  both  information  sought 
and  processed  and  the  perceived  importance  of 
accurate  estimates.  In  the  short  term,  however, 
the  personality  characteristics  defined  as  con¬ 
stituting  managerial  talent  would  probably  be 
relatively  constant  for  a  specific  individual 
and  would  certainly  be  difficult  for  management 
to  influence.  Thus  the  relationships  with  man¬ 
agerial  talent,  though  important  for  other  pur¬ 
poses,  have  little  relevance  to  this  paper. 
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CONCLUSIONS 

Estimating  the  duration  of  an  activity  in  a  net¬ 
work  management  system  is  a  situation  fraught 
with  risk  and  uncertainty  at  best.  The  estimate 
is,  in  reality,  only  the  estimator's  knowledge 
and  experience  expressed  as  a  numerical  guess. 
The  level  of  uncertainty  should  be  directly  re¬ 
lated  to  the  estimator's  familiarity  with  the 
task  and  the  frequency  with  which  it  has  been 
performed  before.  Particularly  in  advanced 
R  &  D  efforts  where  many  of  the  activities  are 
being  performed  for  the  first  time,  absolute 
accuracy  is  impossible  to  obtain.  The  studies 
summarized  in  this  paper  demonstrate,  however, 
that  a  few  fairly  simple  managerial  actions  can 
control  and  reduce  the  level  of  uncertainty  in¬ 
herent  in  most  projects. 

The  first  study  offers  the  project  manager  two 
clues  to  improving  the  accuracy  of  his  project's 
estimates.  The  first  involves  subjectively 
evaluating  the  overall  level  of  technical  un¬ 
certainty  inherent  in  his  project.  If  his  pro¬ 
ject  involves  a  moderate  to  low  level  of  techni¬ 
cal  uncertainty,  then  the  manager  cannot  expect 
to  obtain  significantly  improved  estimates  as 
the  project  progresses.  That  is,  no  "learning 
curve"  is  likely  to  exist.  Apparently,  similar 
activities  to  those  performed  in  this  type  of 
project  have  been  performed  frequently  before, 
and  ost imators  have  a  good  base  of  knowledge  to 
work  from.  Uncertainties  in  these  projects  ap¬ 
parently  stem  from  the  truly  uncontrollable 
factors.  The  project  manager  would  probably 
profitably  reduce  his  efforts  to  obtain  updated 
estimates.  On  the  other  hand,  projects  with 
moderate  to  high  levels  of  technical  uncertainty 
are  likely  to  experience  an  estimating  learning 
curve  as  more  information  becomes  known  about 
the  technical  problems  being  encountered.  Thus, 
later  estimates  can  be  given  increased  credibil¬ 
ity.  Second,  the  most  accurately  estimated 
activities  in  all  projects  (in  terms  of  ACC  #1) 
are  those  lasting  six  weeks  to  two  months. 

Longer  duration  activities  tended  to  be  severely 
underestimated,  yet  many  networks  for  major 
weapon  system  development  programs  contain  ac¬ 
tivities  estimated  to  last  six  to  nine  months 
or  longer.  The  project  manager  could  readily 
divide  such  lengthy  activities  into  several 
shorter  activities  for  estimating  purposes,  in¬ 
creasing  the  level  of  detail  in  the  network. 

This  is  the  same  philosophy  used  in  creating  the 
network  for  the  project  in  the  first  place. 

This  research  indicates  that  carrying  that  phi¬ 
losophy  on  to  the  point  of  estimating  no  more 
than  approximately  two-month  periods  could  im¬ 
prove  the  accuracy  of  the  estimates. 

Study  II  demonstrated  clearly  that  it  is  possi¬ 
ble  to  improve  the  accuracy  of  activity  esti¬ 
mates  relative  to  those  obtained  by  the  usual 
organizational  methods.  Over  65  percent  of  the 
variance  in  estimating  accuracy  was  esplained 
by  information  sought  and  processed,  a  variable 


which  in  turn  was  strongly  influenced  by  the 
perceived  importance  of  accuracy.  In  the  or¬ 
ganizations  studied,  no  formal  feedback  system 
existed  to  inform  the  estimator  after  the  fact 
about  the  accuracy  of  his  estimates.  As  a 
minimum,  this  makes  it  extremely  difficult  for 
the  estimator  to  improve  his  performance,  or 
even  to  know  if  and  when  he  has  done  so.  Cer¬ 
tainly  individuals  who  develop  a  reputation  for 
consistently  poor  estimates  see  their  job  secur¬ 
ity  and  promotion  potential  reduced  by  their 
poor  performance,  but  it  is  difficult  to  under¬ 
stand  how  this  reputation  would  develop  and  how 
it  could  be  verified.  Note  also  that  this  is 
in  the  form  of  a  negative  reward.  There  was 
apparently  no  system  for  rewarding  those  who 
estimated  well.  The  key  to  improving  estimates 
in  this  environment  apparently  lies  in  con¬ 
vincing  the  estimator  of  how  important  it  is  to 
him,  to  his  career,  and  to  his  organization  that 
his  estimates  be  accurate.  One  essential  ele¬ 
ment  of  this  is  to  document  how  accurate  he 
actually  was,  and  to  differentially  reward  the 
good  and  poor  estimators.  Project  managers 
could  easily  establish  such  a  feedback  system, 
given  the  network  records  already  in  existence. 
Significant  improvements  in  estimating  accuracy 
would  likely  result. 

In  sum,  these  studies  have  demonstrated  that 
uncontrollable  factors  play  a  much  smaller  role 
in  determining  the  accuracy  of  activity  esti¬ 
mates  than  was  previously  believed,  either  by 
project  managers  or  by  the  estimators  them¬ 
selves.  Project  unique  variables,  in  particu¬ 
lar  the  level  of  technical  uncertainty,  do  play 
a  more  vital  role  in  determining  estimating 
accuracy  than  previous  researchers  considered. 
Certainly  much  more  research  is  needed  to  define 
these  project  unique  variables,  and  an  easy-to- 
use  quantifiable  scale  for  measuring  a  pro¬ 
ject's  level  of  technical  uncertainty  would  be 
of  particularly  great  benefit.  Nevertheless, 
even  with  the  limitations  of  current  knowledge, 
these  studies  have  shown  that  relatively  simple 
and  inexpensive  project  management  actions  can 
improve  estimating  accuracy  and,  to  some  extent 
at  least,  control  the  level  of  uncertainty  in 
activity  and  project  estimates. 

BIBLIOGRAPHY 


(1)  Abernathy,  W.J.,  "Subjective  Estimates 
and  Scheduling  Decisions",  Management 
Science,  Vol.  18  (October  1971),  p.  B80- 
B88. 

(2)  Adams,  J.  R.  ,  and  L.  A.  Swanson,  "Infor¬ 
mation  Processing  Behavior  and  Estimating 
Accuracy  in  Operations  Management",  Acade- 
my  of  Management  Journal,  Vol.  19  (March 
1976),  pV  98-110. 


168 


(3)  Adams,  J.R.,  S.E.  Barndt,  and  F.P.  Law¬ 
rence,  "Time  Estimating  Accuracy  of  Non- 
Repetitive  Production  Activities",  37th 
Academy  of  Management  Proceedings,  Or¬ 
lando,  Florida  (August  14-17,  1977), 

p.  164-168. 

(4)  Archibald,  R.D.,  and  R.W.  Villoria,  Net¬ 
work.  Based  Management  Systems  (PERT/CPM) 
(New  York:  Wiley,  1967). 

(5)  Elmaghraby,  S.E.,  Activity  Networks:  Pro¬ 
ject  Planning  and  Control  by  Network 
Models  (New  York:  Wiley,  1977),  p.  258- 
308 . 

(6)  Chiselli,  E.E.,  Explorations  of  Managerial 
Talent  (Pacific  Palisades,  California: 
Goodyear ,  1971) . 

(7)  Hall,  D.T.,  and  E.E.  Lawler,  "Job  Char¬ 
acteristics  and  Pressures  and  Organiza¬ 
tional  Integration  of  Professionals", 
Administrative  Science  Quarterly,  Vol  15 
(September  1970),  p.  271-281. 

(8)  Kahn,  R.L.,  et  al.,  Organizational  Stress: 
Studies  in  Role  Conflict  and  Ambiguity 
(New  York:  Wiley,  1964)  . 

(9)  Kidd,  J.P.,  and  J.R.  Morgan,  "The  Use  of 
Subjective  Probability  Estimates  in  As¬ 
sessing  Project  Completion  Times", 
Management  Science,  Vol.  16  (November 
1979),  p.  266-269. 

(10)  King,  W.R.,  D.M.  Wit tevrongel ,  and  K.D. 
Hezel,  "On  the  Analysis  of  Critical  Path 
Time  Estimating  Behavior",  Management 
Science,  Vol.  14  (September  1967), 

p.  79-84. 

(11)  King,  W.R.,  and  P.A.  Lukas,  "An  Experi¬ 
mental  Analysis  of  Network  Planning", 
Management  Science,  Vol.  19  (August  1973), 
p.  1423-1432. 

(12)  King,  W.R. ,  and  R.A.  Wilson,  "Subjective 
Time  Estimates  in  Critical  Path  Planning- 
A  Preliminary  Analysis",  Management 
Science,  Vol.  13  (January  1967),  P.  307- 
320. 

(13)  Klingle,  A.R.,  Jr.,  "Bias  in  PERT  Pro¬ 
ject  Completion  Time  Calculations  for  a 
Real  Network",  Management  Science,  Vol. 

13  (December  1966) ,  p .  B194-B201 . 

(14)  MacCrimmon,  K.R. ,  and  C.A.  Ryavec,  "An 
Analytical  Study  of  the  PERT  Assumptions", 
Operations  Research,  Vol.  12  (1964), 

p.  16-37. 

(15)  McClelland,  D.C.,  "That  Urge  to  Achieve", 
in  Readings  in  Management,  ed .  Richards 
and  Nielander  (Chicago:  Southwestern, 
1969),  p.  579-583. 


(16)  Meyers,  S.M.,  "Who  Are  Your  Motivated 
Workers",  Harvard  Business  Review,  Vol.  42 
(January-February,  1964) ,  p”  73-88 . 

(17)  Woodgate,  H.S.,  Planning  by  Network 

(New  York:  Brandon/Systems  Press,  1967). 


169 


A  STATE-OF-THE-ACQUISITION-PROCESS  EVALUATION 

* 

Michael  D.  Rich 
The  Rand  Corporation 


This  paper  relies  primarily  on  results  of  a 
recent  Rand  study  led  by  Edmund  Dews  and  Giles 
Smith.  The  summary  and  findings  of  that 
study**  follow: 


STUDY  OBJECTIVE  AND  APPROACH 


The  primary  objective  of  this  study  is  to 
assist  the  Office  of  the  Secretary  of  Defense 
in  its  current  reassessment  of  defense  acqui¬ 
sition  policy  by  providing  some  quantitative 
insights  into  the  effectiveness  of  the  policy 
changes  adopted  at  the  beginning  of  the  1970s 
at  the  initiative  of  David  Packard,  then  Deputy 
Secretary  of  Defense.  Related  objectives  are 
(1)  to  identify  policy  areas  where  new  initia¬ 
tives  seem  desirable  or  further  research  would 
be  profitable,  and  (2)  to  provide  a  set  of 
organized,  quantitative,  cross-program  data  as 
a  basis  for  future  studies  and  comparisons. 

The  approach  emphasizes  quantitative  analysis. 
The  principal  source  of  data  is  the  Selected 
Acquisition  Reports  (SARs)  issued  quarterly  for 
each  major  defense  system  being  acquired.  The 
most  recent  SARs  analyzed  in  detail  here  are 
those  for  March  1978.  Of  the  total  of  nearly 
60  major  systems  now  reported  on  in  the  SARs, 
some  30  were  selected  for  study  as  being  most 
representative  of  1970s  experience  under  the 
Packard  guidelines.  Among  the  systems  excluded 
were  those  that  had  already  entered  full-scale 
development  before  1969  and  therefore  presum¬ 
ably  reflected  earlier  acquisition  policies. 

The  report  addresses  five  main  questions: 


o  Has  there  been  a  positive  response  to  the 
new  policy  guidelines  established  early 
In  Liu*  1970s? 

o  How  have  I  lie  resulLs  achieved  in  the  1970s 
acquisition  programs  compared  with  the 
goals  established  at  the  time  the  programs 
entered  full-scale  development? 
o  In  terms  of  'these  result-to-goal  compar¬ 
isons,  are  tfie  1970s  programs  doing  better 
than  the  1960s  programs? 
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o  Is  it  now  taking  longer  to  develop  and 
field  systems  than  it  did  in  the  past? 
(The  comparison  is  limited  to  fixed-wing 
aircraft . ) 

o  What  new  initiatives  and  further  research 
are  suggested  by  these  quantitative 
results? 


The  answers  to  these  questions  constitute  the 
major  findings  of  the  study  and  are  summarized 
below. 


RESPONSE  TO  THE  PACKARD  POLICY  INITIATIVES 


Of  the  10  major  elements  in  Mr.  Packard’s 
policy  initiatives,  6  led  to  positive  changes 
in  organizational  structure  or  standard  oper¬ 
ating  procedures:  (1)  the  Defense  Systems 
Acquisition  Review  Council  (DSARC)  was  estab¬ 
lished  to  provide  systematic,  high-level  pro¬ 
gram  reviews;  (2)  the  Cost  Analysis  Improvement 
Group  (-CAIG)  was  established  to  provide  OSD 
with  independent  cost  estimates;  (3)  "design-to- 
cost"  was  instituted,  with  a  specific  cost  goal 
identified  as  a  major  program  objective  for 
each  system;  (4)  responsibility  for  operational 
test  and  evaluation  was  shifted  from  the  devel¬ 
oping  agencies  to  other,  independent  commands; 
(3)  training  courses  and  schools  were  estab¬ 
lished  to  prepare  officers  for  program  manage¬ 
ment;  and  (6)  program  managers  were  given 
written  charters  as  a  means  of  establishing 
their  authority. 

The  remaining  four  elements  required  more  dis¬ 
cretionary  responses,  often  involving  program- 
by-program  decisions  at  Service  level;  these 
responses  were  examined  using  a  quantitative 
approach . 

Our  quantitative  analysis  of  program  rrhruiji'r 
qualifications  suggests  a  trend  in  the  direc¬ 
tion  of  better-qualified  managers,  but  the 
results  depend  on  limited  data  and  are  general¬ 
ly  not  statistically  significant.  From  inter¬ 
views  and  other  qualitative  evidence  we  con¬ 
clude  that  most  program  managers  are  now 
reasonably  well  qualified  for  the  job,  and  some 
are  very  well  qualified  indeed.  Compared  with 
other  groups  for  which  data  were  available  on 
promotion  experiences  program  managers  appear 
to  have  done  very  well  on  the  promotion  ladder 
in  recent  years,  but  questions  can  be  raised 
about  the  composition  of  the  groups  compared, 
and  we  regard  the  results  as  suggestive  rather 
than  conclusive.  Because  of  the  inconclusive 
nature  of  these  results,  and  because  of  the 
widely  divergent  views  expressed  by  program 
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managers  and  other  program  personnel  about  pro¬ 
gram  management  as  a  Service  career,  we  believe 
OSD  and  the  Services  should  not  relax  their 
attempts  to  attract  superior  officers  to  pro¬ 
gram  management  through  favorable  promotion 
opportunities  and  other  incentives. 

Job  tenures  for  program  managers  have  clearly 
been  increasing,  as  called  for  in  OSD  policy, 
and  are  now  between  2  1/2  and  3  years  on  the 
average;  but  the  increase  had  begun  by  the  mid- 
1960s,  well  before  the  new  guidelines  were 
established.  Length  of  tenure  may  now  be  in 
the  right  ballpark,  but  guidance  may  be  needed 
concerning  the  timing  of  program  manager  as¬ 
signments  so  as  to  coincide  with  natural  break¬ 
points  in  program  evolution. 

The  call  for  early  hardware  testing  has  had  a 
strong  positive  response.  Testing  prior  to 
both  DSARC  Milestone  II  (approval  for  full- 
scale  development)  and  DSARC  Milestone  III 
(approval  for  production)  increased  markedly 
during  the  1970s,  so  that  by  1978  the  hard  data 
available  at  major  decision  milestones  was  much 
greater  than  it  had  been  previously.  The  call 
for  a  decrease  in  development /production  con¬ 
currency  has  also  been  answered,  as  shown  by 
the  high  percentage  of  performance  goals  now 
achieved  before  DSARC  Milestone  III. 

The  response  to  Mr.  Packard's  call  for  in¬ 
creased  use  of  hardware  competition  during 
development  has  also  been  positive,  but  not  so 
clearly  marked  as  in  the  case  of  hardware  test¬ 
ing.  About  two-thirds  of  the  programs  that 
have  reached  DSARC  Milestone  II  since  1973 
involved  significant  use  of  hardware  competi¬ 
tion  either  before  they  entered  full-scale 
development  or  subsequently.  This  change  from 
the  situation  in  the  1960s,  when  hardware  com¬ 
petition  in  development  was  rare,  was  achieved 
in  part  because  of  the  Advanced  Prototyping 
Program,  which  provided  direct  dollar  incen¬ 
tives  for  the  Services  to  opt  for  an  acquisi¬ 
tion  strategy  involving  hardware  competition. 
However,  for  some  programs  that  reached  Mile¬ 
stone  II  in  1976  and  afterward,  favorable 
opportunities  for  hardware  competition  may  not 
have  been  exploited.  The  Advanced  Prototyping 
Program  has  not  been  continued,  and  there  is  as 
yet  no  strong  commitment  to  hardware  competi¬ 
tion  in  OSD's  formal  policy  documents,  although 
there  is  a  cross-reference  to  0MB' s  Circular 
A- 109.  The  future  of  this  key  element  of  the 
Packard  initiatives  therefore  appears  somewhat 
in  doubt,  and  a  strong  affirmation  of  OSD's 
commitment  to  hardware  competition  may  be 
desirable,  especially  in  view  of  the  superior 
cost-growth  record  (discussed  below)  of  the 
programs  with  hardware  competition. 

On  balance,  all  policy  elements  being  consider¬ 
ed,  the  Packard  guidelines  appear  to  have  been 
generally  complied  with.  The  result  is  an 
acquisition  environment  in  the  1970s  sub¬ 
stantially  different  from  that  of  the  1960s. 


1970s  EXPERIENCE:  PERFORMANCE,  SCHEDULE, 
AND  COST 


In  comparing  performance,  schedule,  and  cost 
results  with  goals  for  the  1970s  acquisition 
programs,  the  metric  used  was  the  ratio  of  re¬ 
sults  and  goals,  arranged  so  that  in  all  cases 
the  preferred  outcome — higher  performance, 
shorter  schedule,  lower  cost — was  represented 
by  a  ratio  less  than  unity.  The  goals  are 
those  established  at  DSARC  Milestone  II  when 
systems  are  approved  for  full-scale  development. 
The  results  are  those  reported  in  the  SARs 
through  March  1978.  The  aggregate  outcomes  for 
the  programs  examined  were  as  follows: 


o  For  system  performance  parameters ,  the 
distribution  of  ratios  was  nearly  symmet¬ 
rical  around  unity,  with  a  range  from 
about  0.5  to  2.1,  and  a  mean  ratio  of  1.0. 
On  the  average,  performance  goals  were 
achieved  for  the  parameters  tested, 
o  For  scheduled  program  events  accomplished, 
the  distribution  of  ratios  was  skewed 
slightly  toward  higher  values  (schedule 
slippage),  with  a  range  from  about  0.8  to 
2.1,  and  a  mean  ratio  of  1.13.  (These 
ratios  reflect  mainly  experience  in  full- 
scale  development,  because  the  schedules 
established  at  DSARC  Milestone  II  are 
heavily  weighted  toward  development  events 
and  events  early  in  the  production  phase.) 
o  For  program  costs  as  projected  in  March 
1978,  the  distribution  of  ratios  was 
skewed  moderately  toward  higher  values 
(cost  growth),  with  a  range  from  about  0.7 
to  2.2,  and  a  mean  ratio  of  1.20.  The 
dollar-weighted  mean  ratio  was  1.14,  and 
the  median  ratio  was  1.06.  Thus  more  than 
half  of  the  programs  had  cost  growth  of 
less  than  10  percent .  (In  these  compari¬ 
sons,  costs  are  calculated  for  the  pro¬ 
duction  quantity  planned  at  DSARC  Mile¬ 
stone  II  and  are  adjusted  to  eliminate  the 
effects  of  inflation.) 


Cost-growth  ratios  of  the  size  found  here  for 
defense  programs  appear  to  be  in  the  same  ball¬ 
park  as  the  cost-growth  ratios  observed  for 
large  nondefense  projects  involving  new  tech¬ 
nology  or  other  substantial  uncertainties,  al¬ 
though  further  research  is  needed  to  confirm 
this  conclusion. 

The  sample  of  programs  involving  substantial 
hardware  competition  during  or  before  start  of 
full-scale  development  was  characterized  by  con¬ 
siderably  lower  cost  growth  than  the  sample 
without  hardware  competition  (a  cost-growth 
ratio  of  1.16  compared  with  one  of  1.53).  The 
sample  with  hardware  competition  also  did  some¬ 
what  better  in  terms  of  program  schedules  and 
system  performance  goals.  The  only  program  to 
pass  DSARC  III  with  negative  cost  growth  (the 
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UH-60)  had  full  prime  contractor  competition 
through  full-scale  development.  Although  these 
samples  are  small,  this  result  suggests  that 
hardware  competition  deserves  further  attention, 
if  only  to  identify  more  clearly  the  conditions 
in  which  it  is  likely  to  be  advantageous. 

As  programs  mature,  the  projected  constant- 
dollar  cost  to  complete  them  tends  to  increase, 
as  might  be  expected.  No  program  in  our  cost 
analysis  sample  of  31  programs  had  reached  full 
term  completion,  but  17  had  passed  DSARC  II  by 
more  than  three  years.  For  these  17  more 
mature  programs,  the  mean  cost-growth  ratio  was 
1.34  compared  with  1.20  for  the  whole  sample 
including  the  younger  programs.  The  average 
(linear)  rate  of  cost  growth  for  both  the 
mature  sample  and  the  full  sample  was  between 
5  and  6  percent  per  annum.  (This  is  somewhat 
greater  than  the  annual  cost-growth  rates 
recently  calculated  by  the  Office  of  the 
Assistant  Secretary  of  Defense  (Comptroller), 
but  the  calculations  are  for  different  samples, 
and  the  OSD  results  are  expressed  in  terms  of 
compound  rather  than  linear  growth  rates.) 

Apart  from  inflation  and  changes  in  quantity, 
the  major  drivers  of  cost  growth  for  the  pro¬ 
grams  of  the  1970s  were  schedule  changes, 
engineering  changes,  and  estimating  errors. 

For  the  full  31  program  cost  analysis  sample, 
schedule  changes  alone  contributed  about  40 
percent  of  the  total  cost  growth,  or  about  $5 
billion.  There  is  a  clear  need  to  understand 
more  concerning  the  underlying  causes  of 
schedule  change. 

The  record  strongly  suggests  that  a  substantial 
part  of  the  cost  growth  is  not  within  the  area 
of  control  and  responsibility  of  program  mana¬ 
gers,  and  in  some  cases  it  is  even  beyond  the 
scope  of  control  measures  available  to  top 
level  acquisition  managers  in  the  Services  and 
OSD.  Obviously  this  has  important  implications 
for  OSD  acquisition  policy,  and  suggests  that 
the  search  for  better  cost  control  should  in¬ 
clude  consideration  of  changes  in  government 
policy  and  procedures  outside  the  Department 
of  Defense. 

The  conventional  wisdom  is  that  when  programs 
experience  difficulties,  cost  is  the  first  con¬ 
straint  relaxed  and  schedule  the  second,  but 
that  performance  goals  are  adhered  to  more 
rigorously.  For  1970s  experience,  this  view  is 
supported  by  an  examination  of  the  result-to- 
goal  ratios  summarized  earlier.  But,  for  the 
1970s  at  least,  it  must  be  added  that  con¬ 
straints  are  relaxed  (cost  increases  are 
accepted)  for  unit  costs  but  not,  generally, 
for  total  program  costs.  In  the  aggregate, 
total  program  costs  in  constant  dollars  have 
remained  very  close  to  the  amounts  projected  in 
the  Development  Estimates  (DEs)  made  at  the 
time  the  programs  entered  full-scale  develop¬ 
ment.  For  the  31  programs  in  our  cost  analysis 
sample,  reductions  in  quantity  almost  precisely 


canceled  out  the  sum  of  the  cost  changes  due  to 
the  other  variance  categories.  In  other  words, 
the  real  flexibility  in  the  acquisition  process 
is  found  in  the  quantities  of  units  procured, 
not  in  the  aggregate  cost  of  acquisition 
programs . 

This  kind  of  flexibility  raises  important  ques¬ 
tions  about  the  validity  of  the  procurement 
quantities  established  in  the  requirements  pro¬ 
cess  and  the  manner  in  which  quantity-quality 
tradeoffs  are  made. 


1970s  AND  1960s  COMPARISONS 


In  terras  of  the  degree  to  which  program  results 
approach  program  goals,  the  sample  of  1970s 
programs  shows  improvement  over  the  1960s 
sample. 

The  1970s  programs  are  achieving  their  perfor¬ 
mance  and  schedule  goals  to  at  least  the  same 
degree  as  the  1960s  programs  did,  and  are 
probably  doing  slightly  better. 

The  1970s  programs,  moreover,  are  coming  closer 
to  their  cost  goals  by  some  10  to  20  percentage 
points.  (The  calculation  is  in  terms  of  con¬ 
stant-year  dollars  and  DSARC  Milestone  II  pro¬ 
duction  quantities.)  This  is  a  substantial 
reduction  in  cost  growth.  For  the  31  1970s 
programs  in  our  cost  study  the  dollar  sum 
corresponding  to  percentages  of  this  magnitude 
would  be  from  9  to  IB  billion  1979  dollars. 
Cost-growth  avoidance  is  of  course  not  the  same 
as  cost  savings,  but  substantial  cost  savings 
are  implied. 

The  average  annual  linear  rate  of  program  cost 
growth  is  also  less — a  rate  of  about  5  to  6 
percent  in  the  1970s  compared  with  7  to  8  per¬ 
cent  in  the  1960s. 

In  this  comparison  of  acquisition  experience  in 
the  two  decades,  some  caveats  must  bo  borne  in 
mind:  the  somewhat  different  maturities  of  the 

1960s  and  1970s  samples,  the  possibility  of 
differences  in  program  technical  difficulty, 
and  the  influence  of  factors  apart  from  OSD 
policy  and  beyond  the  control  of  program  man¬ 
agement,  for  example,  the  much  higher  rate  of 
inflation  in  the  1970s.  Nonetheless,  we  find 
it  plausible  that  the  changes  in  acquisition 
strategy  and  management  introduced  since  1969 
have  been  the  main  contributors  to  the  observed 
improvements.  If  the  1970s  programs  had  not 
suffered  from  the  unusually  high  rate  of  in¬ 
flation  they  experienced,  these  improvements 
might  well  have  been  greater. 
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ACQUISITION  INTERVALS:  A  SLOWDOWN  IN  THE 
DEVELOPMENT /PROCUREMENT  PROCESS? 


A  recent  study  by  the  Defense  Science  Board 
identified  lengthening  acquisition  intervals 
(slower  fielding  rates)  as  a  critical  defense 
issue.  The  DSB  concluded  that  the  times  re¬ 
quired  for  full-scale  development  had  not 
changed  appreciably,  but  that  there  had  been 
some  lengthening  in  the  early  phase  of  the 
acquisition  process,  before  DSARC  Milestone  II, 
and  also  in  the  production  phase,  after  DSARC 
Milestone  III. 

Because  of  the  importance  of  this  issue,  we 
examined  trends  in  aircraft  fielding  times, 
using  a  data  base  developed  at  Rand  in  connec¬ 
tion  with  earlier  studies.  The  sample  included 
34  U.S.  aircraft  acquired  over  a  period  of 
about  30  years.  We  lacked  good  data  for  the 
front  end  of  the  acquisition  process,  and 
therefore  examined  the  time  trends  only  for 
full-scale  development  (FSD)  and  production. 

The  trend  lines  differed  markedly  for  these 
two  phases  of  the  acquisition  process. 

The  time  taken  to  move  from  the  start  of  devel¬ 
opment  to  first  flight  has  changed  little  over 
the  last  30  years,  perhaps  increasing  very 
slightly.  Total  development  time  (measured 
from  the  start  of  development  to  the  delivery 
of  the  first  production  item)  also  appears  to 
have  changed  little  (for  the  fighters  in  the 
sample) ,  or  even  to  have  decreased  somewhat 
(for  the  larger  sample  including  bombers  and 
transport  aircraft).  These  results  appear 
roughly  consistent  with  the  conclusions  of  the 
Defense  Science  Board. 

The  production  phase,  however,  is  taking  much 
longer  than  it  used  to,  as  measured  by  the  time 
between  the  delivery  of  the  first  and  the  200th 
unit;  this  interval  more  than  doubled  in  the 
course  of  30  years.  Again,  this  result  is 
consistent  with  the  DSB’s  findings.  The  cause 
of  the  lowered  production  rate  is  apparently 
fiscal  rather  than  technical:  higher  produc¬ 
tion  rates  are  generally  quite  feasible  in 
terms  of  manufacturing  capabilities  and  are 
often  planned,  but  program  funding  rates  for 
production  have  failed  to  keep  paee  with  the 
increasing  unit  costs.  The  trend  line  for  air¬ 
craft  investment  rates  (constant-year  dollars 
expended  per  month  for  the  procurement  of  air¬ 
craft  in  the  production  phase)  has  remained 
almost  level  over  time. 

Even  with  the  marked  increase  in  production 
times,  the  net  effect  of  the  different  trends 
in  the  successive  phases  of  the  acquisition 
process  has  been  only  a  modest  increase  in 
total  fielding  times.  The  interval  between 
the  start  of  development  and  the  delivery  of 
the  200th  production  item  has  increased  by  less 
than  10  percent  over  the  30-year  period — an 
average  linear  rate  of  increase  of  only  a 


fraction  of  one  percent  per  year.  This  does 
not,  as  explained  earlier,  take  into  account 
any  lengthening  that  may  be  occurring  in  the 
pre-Milestone  II  phases  of  the  acquisition 
process. 

The  results  just  summarized  refer  to  a  sample 
that  excludes  three  recent  aircraft  programs 
each  characterized  by  a  distinct  prototype 
phase  preceding  DSARC  Milestone  II — the  A-10, 
the  F-16,  and  the  F-18.  These  aircraft  were 
excluded  from  the  trend  analysis  because  of  a 
conceptual  problem  concerning  the  proper  timing 
for  the  start  of  development.  Should  Milestone 
II  be  the  baseline  date,  or  is  it  more  realis¬ 
tic  in  these  three  programs  to  consider  devel¬ 
opment  as  beginning  earlier  with  the  initiation 
of  the  prototype  phase? 

For  these  aircraft  we  examined  both  data 
points.  If  the  development  phase  baseline  is 
dated  from  the  initiation  of  the  prototype 
phase,  the  data  points  lie  above  the  trend 
lines  and  thus  suggest  a  continuing  (or  possi¬ 
bly  accelerating)  increase  in  total  fielding 
times.  If  DSARC  Milestone  IT  Ik  regarded  as 
the  correct  development  baseline,  the  data 
points  for  these  aircraft  generally  fall  below 
the  trend  lines  and  thus  suggest  either  a 
reversal  of  the  trend  toward  longer  total 
fielding  times,  or  some  reduction  in  the  his¬ 
torical  rate  of  increase. 


SUGGESTED  POLICY  INITIATIVES  AND  TOPICS  FOR 
ADDITIONAL  RESEARCH 


Improve  the  Acquisition  Information  Data  Base 


Any  systematic  attempt  to  improve  acquisition 
policy  should  be  supported  by  an  equally  sys¬ 
tematic  attempt  to  improve  the  quality  and 
extent  of  program  data.  The  Selected  Acquisi¬ 
tion  Reports  already  represent  a  major  im¬ 
provement  in  program  data  tracking  compared 
with  what  was  available  before  they  were  ini¬ 
tiated  in  the  late  1960s.  However,  because  of 
their  specialized  and  limited  focus,  the  SARs 
are  not  a  fully  satisfactory  source  of  data  for 
analysis  of  broad  acquisition  policies. 

A  policy-oriented  data  base  should  be  estab¬ 
lished  in  OUSDRE.  Such  a  data  base  could 
utilize  SAR  information  but  should  go  beyond 
the  present  SARs  in  at  least  two  areas.  First, 
original  baselines  should  be  retained  through¬ 
out  the  life  of  the  program,  together  with  a 
full  documentation  of  all  formally  approved 
program  changes.  To  the  extent  possible,  the 
reasons  for  such  changes  in  approved  program 
goals  should  also  be  recorded  (e.g.,  milestone 
slipped  because  of  budget  reduction,  or  tech¬ 
nical  difficulty)  so  that  cause-effect  rela¬ 
tionships  might  be  established.  Second,  to 
facilitate  comparison  of  cost  growth  among 
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many  programs  on  an  internally  consistent 
basis,  a  different  method  of  calculating  cost 
variances  should  be  used  when  there  are  changes 
in  the  buy  size. 


Reduce  the  Instability  in  Program  Funding 
and  Scheduling 


No  major  acquisition  program  can  be  planned  and 
managed  with  high  efficiency  if  it  faces  fre¬ 
quent  and  unpredictable  changes  in  year-by¬ 
year  program  funding  and  production  scheduling, 
even  if  total  program  funding  eventually 
reaches  the  originally  planned  amount.  Sched¬ 
ule  slippage  and  cost  growth  are  the  closely 
related  and  mutually  reinforcing  effects  of 
program  funding  instability.  According  to  the 
SARs  we  examined,  about  40  percent  of  program 
cost  growth  is  attributable  to  schedule 
changes.  Schedule  changes,  especially  in 
operational  testing  and  production,  are  a 
typical  response  to  changes  in  annual  program 
budgets.  Presumably  a  large — but  undetermined 
— share  of  this  cost  growth  is  therefore  ul¬ 
timately  due  to  funding  instability.  We 
suggest  three  approaches  to  this  problem: 


o  Provide  what  is  now  lacking:  strong  OSD 
policy  guidance  as  to  the  desirability 
and  means  of  reducing  program  budget 
fluctuations  and  schedule  changes.  For 
this  purpose  we  offer  a  draft  policy 
statement  in  Section  VI. 

o  Institute  a  study  of  the  relationship 
between  annual  funding  instability, 
schedule  slippage,  and  cost  growth  to 
quantify  more  definitively  the  effects  of 
annual  budget  fluctuations  on  acquisition 
efficiency. 

o  As  a  part  of  the  policy-oriented  data  base 
discussed  above,  methods  should  be  estab¬ 
lished  for  routinely  collecting  infor¬ 
mation  on  changes  in  program  budgets  and 
the  consequent  changes  in  program  struc¬ 
tures  so  that  the  effects  of  budget 
fluctuations  can  be  more  accurately 
assessed  and  their  causes  identified. 


Strengthen  Guidance  on  Hardware  Competition 
in  Development 

The  evidence  offered  in  Section  III  of  this 
report  presents  at  least  a  ’pvima  facie  case  in 
support  of  Mr.  Packard's  emphasis  on  hardware 
competition.  However,  in  the  latest  OSD  policy 
statements  we  have  seen,  hardware  competition 
receives  little  attention;  the  topic  is  han¬ 
dled  essentially  through  cross-references  to 
0MB  Circular  A-109.  As  the  Advanced  Proto¬ 
typing  Program  has  not  been  continued,  this 
indirect  way  of  stating  policy  can  be  inter¬ 
preted  as  a  lessening  of  emphasis  on  hardware 


competition  before  and  during  full-scale  devel¬ 
opment.  If,  as  we  believe,  this  interpretation 
is  not  intended,  a  partial  solution  can  be 
achieved  by  means  of  a  suitable  statement  in¬ 
serted  in  DoD  Directive  5000.1  and  related 
documents,  affirming  OSD's  commitment  to  com¬ 
petition  beyond  the  paper  proposal  stage. 

More  than  this  affirmation  appears  to  be  needed, 
however.  A  general  prescription  in  favor  of 
competition  where  "beneficial"  or  "practical" 
is  not  enough.  What  is  needed  is  guidance  that 
will  help  the  Services  to  decide  when,  under 
what  circumstances,  for  what  kinds  of  systems 
and  contractors,  and  how  far  into  development 
hardware  competition  appears  desirable.  Guid¬ 
ance  of  this  kind  should  be  based  on  experience. 
This  suggests  a  need  for  a  more  detailed  exam¬ 
ination  of  program  histories  than  could  be 
attempted  in  this  study.  Recent  samples  of 
programs  with  and  without  hardware  competition 
should  be  compared  in  detail. 

Emphasize  Production  Quantity  as  an  Element  in 
the  Requirements  Process 

This  study  did  not  directly  examine  the  require¬ 
ments  process,  but  our  results  suggest  that  at 
the  time  the  need  for  a  new  system  is  estab¬ 
lished  the  probability  of  attaining  the  planned 
production  quantity  may  not  receive  sufficient 
management  attention.  As  has  been  observed 
before  and  confirmed  by  this  study,  system 
performance  goals  and  planned  program  costs  are 
adhered  to  rather  closely  in  the  aggregate. 

For  many  reasons,  however,  acquisition  costs 
per  unit  tend  to  rise  above  the  cost  goals. 

The  eventual  reconcilement  with  near-fixed 
total  program  costs  is  typically  achieved  by 
means  of  a  substantial  decrease  in  production 
quantity.  This  apparent  flexibility  as  to  the 
acceptable  size  of  the  operational  inventory 
raises  questions  about  the  validity  of  the 
original  requirement  and  suggests  that  produc¬ 
tion  quantity  and  quantity-quality  tradeoffs 
should  receive  greater  emphasis  in  the  re¬ 
quirements  process. 


Continue  Incentives  To  Make  Program  Management 
an  Attractive  Service  Career 


Although  there  are  indications  that  the  status 
of  program  managers  improved  somewhat  during 
the  1970s  and  that  their  promotion  experience 
was  favorable  relative  to  some  other  groups  of 
officers,  the  evidence  is  inconclusive  and 
perceptions  are  mixed.  The  interviews  suggest 
that  many  senior  and  middle-level  officers  now 
in  the  program  management  career  field  still 
have  doubts  about  what  it  has  to  offer. 

Efforts  to  attract  superior  officers  to  program 
management  should  not  be  relaxed. 


* 


* 
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Examine  the  Timing  of  Program 
Manager  Assignments 


Average  job  tenures  for  program  managers  have 
been  steadily  increasing  since  the  mid-1960s 
and  may  now  be  in  the  right  ballpark.  What  is 
less  clear  is  that  program  manager  assignments 
are  individually  well  timed  with  respect  to 
natural  transition  points  in  program  evolution. 
OSD  policy  is  silent  as  to  the  preferred  time 
phasing  of  assignments.  Our  impression  is 
that  there  is  insufficient  understanding  about 
what  constitutes  good  timing  in  terms  of  pro¬ 
gram  needs,  and  that  this  question  deserves 
examination. 


♦ 
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TAXONOMIC  CONCEPTS  PANELS  SUMMARY 

Taxonomic  is  a  derivative  of  the  word  taxonomy, 
meaning  the  study  of  the  general  principles  of 
scientific  classification.  The  first  task  of 
the  panels  discussing  ’’Taxonomic  Concepts"  was 
to  identify  the  key  aspects  of  risk  and  uncer¬ 
tainty  as  they  related  to  major  projects  with¬ 
in  DoD  and  the  civilian  sector.  All  of  the 
panels  in  these  sessions,  in  addition  to  most 
of  the  panels  in  the  entire  workshop,  had  great 
difficulty  with  terminology  not  only  in  a  def¬ 
initional  but  also  a  communicative  sense. 

While  none  of  the  panels  stated  assumptions 
concerning  structural  considerations  or  approa¬ 
ches,  the  level  of  uncertainty  discussed  ranged 
from  certainty,  to  relative  uncertainty,  to 
complete  uncertainty.  This  was  used  as  a 
spectrum  for  defining  risk  and  uncertainty 
based  on  the  amount  of  information  by  Golden 
and  Martin.  However,  they  did  make  the  assump¬ 
tion  that  risk  and  uncertainty  could  be  treated 
synonymously . 

As  a  baseline  to  stimulate  further  thinking  and 
discussion  one  taxonomy  classified  uncertainty 
into  four  classes.  These  were; 

Environmental  Uncertainty 
Functional  Uncertainty 
Informational  Uncertainty 
Technical  Uncertainty 

The  four  classes  were  described,  defined  and 
refined  with  several  subclasses.  For  example, 
the  first  subclass  to  Environmental  Uncertainty 
included  the  following  categories: 

Nature 

Social  and  Political 
Communication  Media 
Time 

Other  subclasses  were  defined  for  Environmental 
Uncertainty,  each  with  several  categories. 

There  were  four  subclasses  for  Functional  Un¬ 
certainty,  each  with  other  categories  that  were 
also  defined  and  discussed. 

There  was  no  order  implied  in  the  classification. 
However,  the  panel  did  suggest  the  possibility 
that  the  primary  structure  might  be  rearranged 
by  the  "degree  of  control"  available  to  the 
decision  maker. 

Another  serious  discussion  dealt  with  the  defi¬ 
nitions  of  the  category  of  Information  Uncer¬ 
tainty.  The  subclasses  were  divided  into 
anticipated  (or  unanticipated)  unknowns,  and 
known  (or  unknown)  unknowns .  Thes-e  were  based 
on  the  effect  the  factors  will  have  on  programs 
versus  the  second  subclass  that  includes 
factors  for  all  uncertainty,  whether  they 
directly  influence  the  project  or  not.  There 
were  questions  as  to  whether  this  category  was 
similar  to  the  others,  or  in  fact  was  a  dif¬ 
ferent  way  of  explaining  or  defining  the  levels 


of  uncertainty.  In  any  event,  the  panel  rec¬ 
ommended  the  combination  of  the  subclasses. 

Another  approach  to  classification  might  be  by 
type  of  phenomena,  or  analysis.  This  could 
range  from  high  to  low  probability,  small  to 
large  variance,  subjective  to  objective  prob¬ 
ability  and  distribution  type. 

A  different  tack  or  method,  described  by 
Thompson,  was  to  categorize  the  uncertainties 
of  the  acquisition  process  using  the  follow¬ 
ing:  the  plan,  the  program  manager,  and  the 

relationship  with  the  environment.  This 
taxonomy  was  not  so  much  a  model  with  the 
causal  characteristics  or  classification 
normally  associated  with  a  taxonomy,  but 
rather  presented  dimensions  to  serve  as  a 
basis  for  the  analysis  of  uncertainty. 

The  less  structured  aspect  of  this  approach 
is  akin  to  what  might  be  called  "blob  theory" 
where  a  field  is  analyzed  from  multiple 
perspectives  paying  particular  attention  to 
the  relationships  between  the  various  per¬ 
spectives.  The  methodology  discussed  in  this 
panel  was  characterized  as  spectaculative 
theorizing  of  an  exploratory  nature  and  used 
an  ordinal  scale.  Such  a  scale  is  easy  to 
define  and  apply  by  an  operating  manager  and 
can  also  be  easily  converted  to  an  interval 
or  ratio  scale  if  necessary. 

The  proposed  taxonomy  for  research  provided 
normative  objectives  and  dealt  with  the 
approaches  that  should  be  applied  to  pro¬ 
curement  and  acquisition  research. 

The  second  objective  of  the  panels  was  to 
describe  the  present  state-of-the-art  in  the 
area  of  risk  and  uncertainty  and  analysis. 
First  and  foremost,  the  paramount  issue  in 
all  of  the  panels  was  the  need  for  agreement 
on  the  definitions  of  risk  and  uncertainty. 
Those  working  in  the  field  of  risk  and  un¬ 
certainty  come  from  diverse  experience  and 
academic  backgrounds  and  thus  have  different 
meanings  for  the  same  terms  and  words. 

With  the  mix  of  participants  (academe,  gover- 
ment  and  industry)  the  need  for  the  identi¬ 
fication  of  differences  from  the  mathematical 
to  the  practical  usage  of  terms  in  discussions 
was  most  apparent  and  frequently  frustrating. 
The  same  terms  were  used  for  many  very  dif¬ 
ferent  concepts  and  methodologies. 

One  model  (Thompson)  is  a  substructure  for 
deterministic  descriptions  and  leads  to  a 
method  for  combining  ordinal  variables  with 
such  items  as  the  product  life  cycle.  In¬ 
dustry  can  be  segmented  using  the  model’s 
dimensions,  particularly  for  the  purpose  of 
looking  at  behavioral  patterns  within  these 
sectors . 

The  state-of-the-art  as  discussed  by  the  panel 
-eviewing  the  estimating  network  analysis 
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activity  paper  (Adams  and  Busch)  provided  in 
formation  of  value  to  project  managers  and 
others  concerned  with  estimating  accuracy.  In 
general,  it  can  be  expected  that  estimators, 
who  have  the  responsibility  to  eventually 
perform  the  task  being  estimated,  may  be 
likely  to  introduce  some  bias  into  the  esti¬ 
mate.  Estimators  are  also  subject  to  influ¬ 
ence  by  management  to  bias  values  in  order  to 
conform  to  specified  targets  and  objectives. 

For  project  activities  which  have  a  "fluid 
completion  date  (e.g.  software  development) 
there  is  a  tendency  for  the  manager  to 
"legislate"  the  end  date  to  be  as  close  as 
possible  to  the  estimator’s  date. 

It  was  obvious  that  the  state-of-the-art  in 
risk  and  uncertainty  is  still  in  the  phase 
of  random  applications.  The  need  for  defin¬ 
ing  categories  and  developing  taxonomies  is 
essential  to  order  and  progress.  For,  with¬ 
out  well  defined  and  understood  terminology 
the  semantics  required  to  translate  or  inte¬ 
grate  applications  into  state-of-the-art 
via  a  taxonomy  and  to  flush  out  the  skeletal 
framework  will  be  extremely  difficult,  if 
not  impossible. 

WhLle  the  panels  were  to  analyze  existing 
taxonomies  of  risk  and  uncertainty  analysis 
for  weakness  and  omissions,  this  was  consi¬ 
dered  in  depth  by  only  one  panel.  Here  a 
suggestion  was  made  for  the  reordering  of 
categories  by  degree  of  control  (noted 
above)  and  the  combination  of  the  subclasses 
of  business  risk  and  financial  risk  into  one 
category.  A  taxonomy,  by  definition,  is  a 
classification  that  in  turn  specifies 
order.  Consideration  should  be  given  how 
to  order  after  general  agreement  is 
achieved  on  categories  and  subclasses. 

There  were  other  weaknesses  and  omissions 
noted  by  most  of  the  panels  in  these  sessions. 
However,  they  dealt  primarily  with  methodology 
so  they  are  not  included  in  this  summary. 

The  final  task  for  the  panel  was  the  descrip¬ 
tion  (or  development)  of  a  comprehensive  tax¬ 
onomy  of  risk  and  uncertainty  analysis  relating 
to  major  projects  (DoD  and  civilian  sectors) . 
For  many  of  the  reasons  noted,  this  was  not 
accomplished  to  the  satisfaction  of  the  work¬ 
shop  participants.  However,  significant 
strides  were  made  in  the  direction  of  iden¬ 
tifying  the  areas  needing  further  research. 

These  panels,  more  than  others  in  the  workshop, 
were  confronted  by  the  need  for  basic  defi¬ 
nitions.  Words  such  as:  risk,  uncertainty, 
risk  analysis,  state-of-the-art,  concurrency, 
risk  assessment,  risk  avoidance,  risk  transfer, 
and  so  on,  need  general  agreement  on  defi¬ 
nition  and  usage. 


recommendations 

Finally,  the  discussions  of  the  participants 
in  the  taxonomic  concepts  panels  provides  in¬ 
sight  and  perspective  into  what  needs  to  ho 
done  to  advance  the  field.  These  recommenda¬ 
tions  ,  not  arrange  by  priority,  included: 

1.  Reexamine  the  definitions  of  risk,  uncer¬ 
tainty  and  related  terms  derived  from  past 
work  in  various  areas  and  disciplines. 

2.  The  continued  development  and  research  for 
a  taxonomy  to  include  risk  management  and 
the  subclasses  of: 

Risk  assessment,  risk  analysis,  risk 
turnover,  risk  avoidance,  etc. 

3.  Study  and  research  risk  analysis  and  risk 
management  as  related  to  the  life  cycle 
phase  or  stages. 

4.  Investigate  the  behavioral  aspects  of 
decision  making  under  uncertainty  in  tne 
acquisition  process.  Consider  the  type  of 
contract  employed  as  one  of  the  important 
var  labl es . 

5.  Use  existing  publications,  training  pro¬ 
grams,  and  educational  programs  to  com¬ 
municate  to  users  and  decision  makers  the 
results  of  the  above  research  efforts. 

This  will  require  translation  into  terras 
meaningful  to  the  users. 

6.  Education  in  risk  analysis  is  needed  by 
project  managers  as  well  as  senior  execu¬ 
tives.  This  should  not  be  limited  only  to 
cost  estimators. 

7.  Mathematical  models  can  provide  rapid 
answers  to  risky  problems  that  can  not 
normally  be  handled  by  other  methods. 

8.  Practically  any  decision  is  in  fact  a 
risk  analysis,  but  the  state-of-the-art 
ranges  from  subjective  analysis  to  so¬ 
phisticated  model  building.  However, 
program  managers  require  a  better  under¬ 
standing  of  the  problem. 

9.  Study  the  behavioral  aspects  of  estima¬ 
ting,  including  the  development  of  guide¬ 
lines  designed  to  minimize  estimator  bias. 

10.  Develop  procedures  and  techniques  for 

evaluating  estimates  under  uncertainty. 
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ABSTRACT 


NETWORK  MODELS 


This  paper  traces  the  development  of  network 
models  for  program  management  under  conditions 
of  risk.  The  history  of  analytical  methods  is 
reviewed  briefly,  and  an  interactive  model  for 
the  simulation  of  complex  development  and 
acquisition  programs  is  described. 

INTRODUCTION 


The  uncertainty  faced  by  the  acquisition  man¬ 
age*  in  today's  complex  research,  development, 
and  acquisition  environment  is  considerable. 

A  new  military  weapon  system  program  may  in¬ 
volve  many  components,  many  developers  and 
suppliers,  and  new  and  untested  technologies. 
There  may  be  uncertainty  at  every  step  of  the 
program — uncertainty  as  to  the  success  or 
failure  of  each  program  element,  uncertainty 
as  to  the  time  required,  and  uncertainty  as  to 
the  cost  of  each  program  element.  The  suc¬ 
cessful  management  of  this  process  requires, 
in  addition  to  a  measure  of  good  fortune  and 
clairvoyance,  first-class  analytical  tools  and 
techniques . 

The  Department  of  Defense  requires  that  risks 
be  assessed  at  each  stage  of  the  development 
and  acquisition  process  and  that  system  per¬ 
formance,  cost,  and  schedule  be  considered  in 
the  light  of  the  risks  involved  [2] .  While 
there  is  no  prescribed  format  for  this  pro¬ 
cess,  a  simplistic  approach  to  the  problem 
might  include  subdividing  the  development  and 
i s  i  l  inn  program  into  clearly  defined  pro¬ 
gram  e I oim*nts,  assessing  the  risk  components 
of  each  program  element,  constructing  a  model 
of  the  acquisition  program,  and  exercising  the 
model  in  some  way  to  gain  insight  into  the 
effects  of  risk  on  program  completion. 

This  paper  considers  the  history  of  analytical 
methods  and  modeling  in  addressing  the  problem 
of  program  management  under  conditions  of 
risk.  Some  developments  in  modeling  are  re¬ 
viewed  briefly,  and  a  computer  routine  which 
permits  interactive  initialization  and  modifi¬ 
cation  of  the  model  of  the  acquisition  process 
is  described. 


There  is  nothing  new  about  network  models  for 
program  management.  Indeed,  a  variety  of  net¬ 
work  models  suitable  for  program  scheduling 
have  been  developed.  Perhaps  the  earliest  of 
these  was  the  Gantt  Chart,  developed  by  Henry 
Gantt  during  World  War  I  [1],  The  Gantt  Chart 
is  still  widely  used  for  planning  and  control 
and  is  the  predecessor  of  many  other  network 
models . 

A  need  for  planning  and  control  techniques  for 
increasingly  complex  programs  and  projects  led 
to  the  development  of  critical  path  analysis 
in  the  1950s — Critical  Path  Method  (CPM) , 
developed  for  DuPont  in  1957  by  James  Kelley 
and  Morgan  Walker  for  the  planning  and  control 
of  complex  construction  projects  [3] ,  and 
Program  Evaluation  and  Review  Technique 
(PERT) ,  developed  in  1958  for  the  Polaris  pro¬ 
gram  by  Eooz,  Allen,  and  Hamilton,  in  conjunc¬ 
tion  with  Lookheed  [4].  These  two  techniques, 
which  differ  slightly  in  detail  but  are  essen¬ 
tially  similar,  identify  the  "critical  path" 
(the  longest  path  through  an  oriented  network) 
and  are  widely  used. 

While  PERT  permits  some  variability  in  the 
activity  times  (optimistic,  most  likely,  and 
pessimistic  time  estimates) ,  neither  technique 
permits  probabilistic  branching  at  nodes,  and 
neither  permits  specification  of  probability 
functions  for  activity  times.  Both  are  essen¬ 
tially  deterministic  procedures  and  are  solved 
in  a  variety  of  ways  using  relatively  simple 
algorithms . 

A  logical  extension  of  critical  path  analysis 
was  the  development  of  probabilistic  network 
models  permitting  probabilistic  branching  at 
nodes  and  some  choice  of  activity  time  dis¬ 
tributions.  Two  such  models  are  Risk  Infor¬ 
mation  for  Schedule  and  Cost  Analysis  (RISCA) , 
developed  for  the  Army  Logistics  Management 
Center  at  Fort  Lee  [6] ,  and  Venture  Evaluation 
and  Review  Technique  (VERT),  developed  by 
Gerald  Moeller  for  the  Army  Armament  Command 
at  Rock  Island  Arsenal  [5] . 

RISCA  and  VERT  are  essentially  similar,  dif¬ 
fering  primarily  in  size  and  generality. 

RISCA  considers  two  system  variables,  time  and 
cost,  and  has  a  limited  assortment  of  built-in 
time  distributions  (constant,  uniform,  tri¬ 
angular,  and  normal).  VERT  considers  three 
variables,  time,  cost,  and  performance,  and 
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has  a  wider  choice  of  built-in  probability 
functions  (constant,  uniform,  triangular,  nor¬ 
mal,  lognormal,  Poisson,  Erlang,  gamma,  beta, 
and  tabulated  distributions) . 

RISCA  is  easy  to  use  and  has  relatively  modest 
memory  requirements.  VERT,  with  its  greater 
flexibility,  is  more  difficult  to  use  and 
requires  substantially  more  memory.  Both  are 
"Monte  Carlo"  simulation  techniques,  executing 
the  network  model  of  the  development  program 
repeatedly  and  presenting  results  in  statisti¬ 
cal  form  at  the  conclusion  of  the  simulation. 

Both  RISCA  and  VERT  were  developed  for  card 
input  and  batch-mode  operation,  and  neither 
was  intended  for  terminal  use.  In  order  to 
enhance  the  utility  of  the  simulation 
approach,  an  interactive  version  of  RISCA  has 
been  developed  at  the  University  of  Dayton 
which  features  fully-prompted  initialization 
and  modification  of  the  network  from  the  ter¬ 
minal  keyboard  and  which  returns  simplified 
results  to  the  terminal  and  complete  output  to 
the  system  printer.  The  interactive  version 
parallels  the  card-input  version  and  is 
written  for  the  UNIVAC  90/80.  Additional 
information  is  available  upon  request  from  the 
author. 


REFERENCES 


1.  Clark,  Wallace.  The  Gantt  Chart.  New 

York:  Ronald  Press,  1922. 

2.  Department  of  Defense  Directive  5000.2. 
"Major  System  Acquisition  Process." 
Department  of  Defense,  Washington,  D.C. 

3.  Kelley,  James  E. ,  and  Morgan  R.  Walker. 
"Critical  Path  Planning  and  Scheduling." 
Proceedings  of  the  Eastern  Joint  Computer 

Conference.  1959. 

4.  Malcolm,  D.  G. ,  J.  H.  Roseboom,  C.  F. 

Clark,  and  W.  Fazar.  "Applications  of  a 
Technique  for  Research  and  Development 
Program  Evaluation."  Operations  Research, 
Vol.  VII,  No.  5.  1959. 

5.  Moeller,  Gerald.  "Venture  Evaluation  and 
Review  Technique."  U.S.  Army  Armament 
Command,  Rock  Island,  Illinois.  1972. 

6.  "Risk  Information  for  Schedule  and  Cost 
Analysis  (RISCA),"  U.S.  Army  Logistics 
Management  Center,  Fort  Lee,  Virginia. 


SUMMARY 


Network  models  play  an  important  role  and  can 
be  of  assistance  to  the  acquisition  manager  in 
dealing  with  the  risks  associated  with  complex 
development  and  acquisition  programs.  A  vari¬ 
ety  of  models  have  been  developed  to  handle 
increasingly  complex  situations,  ranging  from 
Gantt  Charts  and  PERT  networking  to  interactive 
network  simulation  models. 
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AN  ENTROPIC  COST  MODEL  FOR  THE  DEVELOPMENT  CONTRACT' 


Colonel  Martin  D.  Martin 
Air  Force  Business  Research  Management  Center 


INTRODUCTION 

The  early  planning  phases  for  a  new 
weapon  system  are  characterized  by  a  high 
degree  of  technical  and  cost  uncertainty  as 
related  to  the  development  and  production 
of  a  new  weapon  system.  During  these  early 
planning  stages,  cost  estimates  are  formu¬ 
lated  for  the  potential  investment  necessary 
to  acquire  and  maintain  a  system  (9).  As 
knowledge  increases  and  better  information 
becomes  available,  initial  cost  estimates  are 
revised  and  updated  at  various  points  of  the 
acquisition  life  cycle.  There  are  many  tools 
available  to  managers  and  their  staffs  for 
generating  cost  estimates  during  early 
planning  and  development  phases  of  a  weapon 
system.  However,  very  few  of  these  methods 
include  a  formal  teciinique  for  evaluating  the 
magnitude  of  the  uncertainty  associated  with 
the  technical  aspects  of  a  program.  These 
considerations  form  the  basis  of  several 
research  efforts  which  are  the  basic  theme  of 
this  paper.  Each  research  effort  and  its 
results  will  be  described.  Then,  the  impli¬ 
cations  of  these  results  will  be  presented. 

COMPLETED  RESEARCH 

The  series  of  research  effort  conducted 
at  the  Air  Force  Institute  of  Technology 
(AFIT)  evolved  from  an  entropic  cost  model 
formulated  under  Air  Force  sponsorship  at  the 
University  of  Oklahoma  in  1971  (8).  Before 
discussing  the  research  studies,  a  brief 
description  of  the  entropic  cost  model  is  in 
order. 

The  Entropic  Cost  Model: 

To  facilitate  understanding  of  the 
entropic  cost  model,  a  few  points  must  be 
introduced  concerning  the  environment  rela¬ 
tive  to  the  development  contract,  that  is, 
one  awarded  in  the  conceptual  or  validation 
phase  of  the  weapons  acquisition  process. 


From  a  contractual  perspective,  there  are 
two  Key  time  periods  which  relate  to  poten¬ 
tial  cost  growth:  pre-contract  award 
(pre-award)  and  post-contract  award 
(post-award).  (See  Figure  1) 

During  the  pre-award  time  frame  of  the 
contract  life  cycle,  management  is  primarily 
concerned  with  strategy  formulation  to  oeal 
with  the  development  of  a  weapon  system. 

Plans  are  made  to  transform  these  strategies 
into  tactics  to,  if  possible,  preclude  cost 
growth  during  development.  To  accomplish 
these  tasks,  management  has  at  its  disposal 
certain  information  that  should  permit  the 
structuring  of  decisions  at  the  time  of  award 
of  the  pertinent  development  contract.  This 
information  includes  technical  informal  ion, 
cost  estimates,  and  the  results  of  risk 
analysis.  Technical  information  includes 
engineering  estimates  and  feasibility  studies 
conducted  by  the  government  or  government 
contractors.  Cost  estimates  are  usually 
available  from  four  sources:  the  Cost  Analy¬ 
sis  Improvement  Group  (CAIG);  Independent 
Cost  Analysis  (ICA);  the  buying  organization 
or  program  office  ( PO ) ;  and  finally,  defense 
contractors.  An  assessment  of  uncertainty  is 
critical;  however,  rigorous  and  formal  treat¬ 
ment  of  this  element  has  not  been  an  integral 
part  of  management’s  information  system. 

Only  recently  have  serious  efforts  been 
attempted  to  structure  the  process  of  analyz¬ 
ing  the  uncertainty  which  confronts  managers 
responsible  for  the  development  contract  and 
the  related  program  (11). 

During  the  post-award  time  frame  (see 
Figure  1),  management  must  monitor  its  infor¬ 
mation  system  and  take  actions  to  preclude  or 
minimize  cost  growth.  However,  the  informa¬ 
tion  which  makes  up  the  control  base  of  the 
development  contract  has  its  roots  in  the 
pre-award  time  frame.  The  most  critical 
point  in  time  for  model  effectiveness  from  a 
planning  standpoint  is  just  prior  to  contract 
award  ( see  Figure  1) . 


Figure  1 
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The  information  set  relative  to  the  devel¬ 
opment  contract  consists  of  two  subsets:  an 
ordered  set  and  a  set  that  is  disordered 
(lacks  order).  The  ordered  set  is  composed 
of  factors  which  appear  to  have  certain  out¬ 
comes  during  development.  This  ordered  set 
forms  the  oasis  for  the  target  cost  (or  the 
theoretical ly  "most  likely"  cost)  for  the 
development  contract.  The  disordered  set 
relates  to  factors  with  uncertain  outcomes 
and  forms  the  basis  for  cost  growth  during 
the  post-award  time  period.  This  information 
set  is  illustrated  in  Figure  2. 


In  the  entropic  cost  modll,  when  order  is 
maximum  or  at  unity,  the  final  cost  is  cer¬ 
tain  or  known  with  certainty.  However,  when 
lack  of  order  or  some  degree  of  disorder  is 
present,  there  are  several  possible  outcomes 
and  final  cost  is  not  certain.  Therefore, 
uncertainty  relates  to  multiple  outcomes. 

The  purpose  of  the  entropic  cost  model  is  to 
estimate  final  cost  by  using  a  quantitative 
expression  of  the  disorder  or  uncertainty 
concerning  development. 


The  disorder  or  uncertainty,  in  informa¬ 
tion  provides  the  foundation  for  the  entropic 
cost  model.  In  set-theory  terminology,  dis¬ 
order  is  the  complement  of  order  in  the 
information  set.  For  the  entropic  cost 
model,  this  disorder  portion  of  the  infor¬ 
mation  set  is  approximately  equal  to  entropy. 
The  theory  of  entropy  derives  from  thermo¬ 
dynamics.  In  a  physical  system,  tne  amount 
of  disorder  is  a  function  of  its  molecular 
state  at  different  temperatures.  The  concept 
of  entropy  was  extended  to  information  sys¬ 
tems  in  the  development  of  communication  and 
information  theory  by  Shannon  and  Weaver  when 
they  attempted  to  explain  noise  in  such  sys¬ 
tems  (10).  The  concept  is  further  extended 
to  form  the  basis  for  the  entropic  cost  model 
which  is  formulated  as  follows: 


Final  Cost  =  Target  Cost _ 

Order  in  Information 

=  Target  Cost _ _ 

1  -  Disorder  in  Information 

=  Target  Cost 
1  -  Entropy 


The  basic  goal  of  the  first  two  research 
efforts  at  AFIT  was  to  test  the  validity  of 
the  model  using  simulated  data  from  develop¬ 
ment  contracts  and  programs.  The  third 
effort  was  conducted  to  determine  if  a  best 
method  was  available  to  structure  and 
generate  data  to  feed  the  model.  The 
following  paragraphs  provide  a  description 
and  the  results  of  each  effort. 

Initial  Model  Validation  Effort: 

Research  Description.  Glover  and  Lenz 
gathered  data  from  the  Short  Range  Attack 
Missile  (SRAM)  development  program  for  use 
in  testing  the  model  (4).  The  researchers 
reviewed  contract  files  and  program 
documentation  to  determine  the  data 
availability  for  the  test.  During  this 
review,  it  was  found  that  the  data  (element 
weights)  necessary  to  test  the  model  was 
contained  in  source-selection  files. 

However,  these  files  were  not  readily  avail¬ 
able  because  of  their  sensitivity.  Valida¬ 
tion  required  the  creation  of  the  information 
set  in  existence  immediately  after  the  award 
of  the  SRAM  development  contract.  Limited 
access  to  source  selection  records  required 
the  researchers  to  develop  a  method  to  simu¬ 
late  and  quantify  the  uncertainty  of 
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information.  The  method  had  to  meet  the 
requirements  of  research  and  be  potentially 
useful  to  management  to  capture  and  measure 
uncertainty. 

The  work  of  C.  Jackson  Grayson  in  his 
study  of  uncertainty  and  risk  and  its  appli¬ 
cation  to  oil  and  gas  drilling  decisions  was 
known  to  the  researchers  (6).  Jackson's 
technique  relied  on  the  use  of  probability 
statements  by  experts  in  oil  and  gas  explora¬ 
tion  and  drilling  relative  to  success  in 
terms  of  production.  Grayson  considered 
both  individual  and  group  risk  preferences  in 
his  work.  A  similar  approach  based  on  sub¬ 
jective  probabilities  was  perceived  to  be 
applicable  to  the  development  contracting 
environment  in  the  000. 

Consequently,  Glover  and  Lenz  conducted 
an  exhaustive  review  of  the  techniques 
available  for  measuring  subjective  opinions 
of  experts.  The  review  identified  the  DELPHI 
method,  developed  at  the  RAND  Corporation,  as 
the  best  approach  to  be  used  to  simulate  the 
development  contract  award  environment  for 
the  SRaM,  Generally,  the  DELPHI  technique  is 
used  to  solicit  expert  judgment  to  forecast 
what  might  happen  in  the  future.  Glover  and 
Lenz  used  the  structure  and  method  of  DELPHI 
to  simulate  what  happened.  They  used  a 
series  of  controlled  interviews  with  feedback 
directed  at  individuals  who  actually  par¬ 
ticipated  in  SRAM  source  selection.  The 
researchers  solicited  statements  of  proba¬ 
bility  concerning  outcome  measures  of 
unacceptable,  acceptable,  and  exceptional  for 
each  of  nine  (2)  factors  of  concern,  such  as 
range,  speed,  etc. 

Using  this  approach,  Glover  and  Lenz  were 
able  to  identify  some  19,683  possible  factor- 
outcome  combinations  and  a  probability  for 
each.  The  calculation  base  was  the  three 
categories  of  outcome  identified  during  the 
simulation  for  each  of  Jhe  possible  factors. 
Therefore,  there  were  3y  or  19,683  possible 
outcomes  for  the  development  program. 

Entropy  was  then  calculated  as  follows: 

Entropy  =  System  Entropy 
Maximum  Entropy 

19,683 
Z  p.  log  p. 
i  =  1 

log  19,683 

=  0.686 

Where:  0  <_  Entropy  <  1,  and 

p.|  =  probability  of  the  ith 
factor  outcome  (2) . 


Using  the  entropy  value  of  0.686,  an 
estimate  for  total  program  cost  as  follows: 

Target  Cost 
Total  Program  Cost  =  1  -  Entropy 

=  S143.3M 
1  -  0.686 

=  $143. 3M 
0.314 

=  $456. 4M 

The  findings  which  resulted  from  this  effort 
were  considered  significant  relative  to  the 
validation  objective. 

Research  Findings.  The  actual  cost  for  the 
SRAM  development  program  was  $439  million. 

The  estimate  for  this  cost  obtained  by 
applying  the  entropic  cost  model  was  $456 
million.  This  estimate  was  based  on  encoun¬ 
tering  the  worst  possible  cost  conditions 
during  development.  There  were  some  adjust¬ 
ments  to  final  cost  data  based  on  approved 
changes  which  were  not  contemplated  during 
source  selection.  Thus,  the  results  of  the 
study  indicated  that  the  entropic  cost  model 
had  merit  for  use  as  an  estimator  of  develop¬ 
ment  cost.  The  unique  feature  of  the  model 
is  its  inclusion  of  a  variable  for  measuring 
uncertainty  -  entropy.  The  results  of  this 
one  research  effort  were  not  considered  ade¬ 
quate  to  validate  the  model.  Therefore, 
another  team  of  researchers  at  AFIT,  Babiarz 
and  Giedras,  attempted  to  replicate  the 
Glover  and  Lenz  approach  (1). 

Model  Validation  Replication  Effort: 

Research  Description.  Babiarz  and  Giedras 
continued  the  effort  to  test  the  entropic 
cost  model  by  replicating  Glover  and  Lenz's 
work.  The  researchers  selected  the  F-5E 
TIGER  II  Aircraft  Development  Program  as 
their  data  source  for  the  test.  The  proce¬ 
dures  were  identical  to  those  used  by  Glover 
and  Lenz  including  the  use  of  DELPHI  to 
attempt  to  simulate  the  uncertainty  at  the 
time  of  contract  award.  However,  Babiarz  and 
Giedras  obtained  results  that  were  different 
from  those  of  Glover  and  Lenz. 

Research  Findings.  Babiarz  and 
Giedras  found  that  the  application  of  an 
approach  such  as  DELPHI  did  not  provide  evi¬ 
dence  to  substantiate  the  use  of  the  entropic 
cost  model  as  a  forecasting  tool.  Their 
effort  produced  a  final  result  that  was  dif¬ 
ferent  from  the  Glover  and  Lenz's  result. 
Consequently,  they  concluded  that  the  use  of 
an  approach  such  as  DELPHI  leaves  much  to  be 
desired  (1). 
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Using  the  results  of  their  uncertainty 
assessment,  Babiarz  and  Giedras  obtained  sub¬ 
jective  probabilities  for  only  the  "poor"  (or 
unacceptable)  and  "acceptable"  categories  for 
twelve  (12)  factors  of  concern  at  the  time  of 
F-5E  development  contract  award.  Therefore, 
there  were  or  4096  possible  outcomes. 
Entropy  was  then  calculated  as  follows: 

System  Entropy 

Entropy  =  Maximum  Entropy 

4096 

l  Pi  log  p. 

i  =  1 

log  4096 

=  0.910 

Where,  0  <  Entropy  <  1,  and 

p-  =  probability  of  the  ltn 
factor  outcome 

Using  this  result,  total  program  costs 
were  then  calculated  as  follows: 

Total  Program  Cost  =  Target  Cost 
1  -  Entropy 

=  S83.635M 
1  -  0.910 

=  83.635M 
0.09 

-  S929.273M 


The  actual  cost  for  the  development  of 
the  F-5E  was  $39,322  million.  Thus,  the 
model  forecast  of  total  program  cost  was 
10.4  times  greater  than  the  actual  cost  (1, 
81).  Based  on  this  result,  the  researchers 
questioned  the  use  of  a  device  such  as  DELPHI 
to  generate  measurements  of  uncertainty.  The 
problems  encountered  did  not  relate  to  the 
model,  but  measurement  of  uncertainty. 
Although  the  basic  approaches  used  by  the 
researchers  on  both  efforts  were  the  same, 
there  were  also  some  subtle  differences  which 
need  to  be  discussed  before  describing  the 
last  research  effort  which  specifically 
addresses  measuring  uncertainty  (5). 


A  Critique  of  Previous  Efforts: 

Lanclos  critiqued  the  application  of  DELPHI 
to  simulate  uncertainty  at  the  time  of 
contract  award  (7).  Lanclos  treated  the  two 
efforts  as  case  studies  and  assessed  tne  dif¬ 
ferences  between  them.  The  following  is  a 
discussion  of  Lanclos'  study  findings  (7, 
9-11). 

Glover  and  Lenz  selected  four  persons  as 
their  expert  panel  who  were  actually  members 
of  the  SRAM  Source  Selection  Evaluation  Board 
(SSEB).  Further,  Glover  and  Lenz  followed  up 
their  research  effort  by  gaining  access  to 
SRAM  SSEB  files  and  were  able  to  verify  their 
uncertainty  assessments  in  all  but  one  area 
of  concern.  Therefore,  the  team  was  strongly 
confident  in  the  assessment. 

Babiarz  and  Giedras  decided  to  use  the 
DELPHI  approach  from  the  start  of  their 
effort  and  did  not  attempt  to  review  SSEB 
files  for  the  F-5E.  Their  panel  members  were 
not  members  of  the  F-5E  SSEB,  but  had  mana¬ 
gerial  positions  in  the  F-5E  program  office 
during  the  development  of  the  F-5E,  one  of 
which  had  to  drop  out  of  the  assessment  due 
to  a  temporary  duty  assignment,  not  at 
contract  award.  Some  of  the  areas  of  concern 
identified  occurred  as  late  as  two  years  into 
the  F-5E  development  program. 

Having  studied  the  significant  differ¬ 
ences  between  the  first  two  efforts  to  test 
the  entropic  cost  model,  Lanclos  concluded 
that  both  efforts  would  have  been  more 
realistic  if  they  had  been  conducted 
"real-time."  Also,  Baoiarz  and  Giedras' 
choice  of  experts  and  problem  areas  did  not 
give  a  true  representation  of  the  uncertainty 
in  F-5E  development  at  the  time  of  contract 
award.  Finally,  the  validity  of  DELPHI  as  an 
approach  to  measuring  uncertainty  was  not 
proved  or  disproved  by  either  effort. 

However,  its  use  to  structure  the  assessment 
of  uncertainty  does  have  merit  (7,  11-12). 

The  concern  of  measuring  uncertainty  was 
addressed  by  a  third  research  effort  at  AF I T 
in  1976  (5). 

The  Measurement  of  Subjective  Probability: 

Research  Description.  Grayson  and 
Lanclos1  research  objective  was  to  evaluate 
existing  techniques  for  assessing  subjective 
probability  (5,  9-10).  Their  objective  was 
to  propose  an  approach  to  assess  and  measure 
the  uncertainty  which  existed  relative  to  a 
given  weapon  system  development.  Their  study 
addressed  the  following  research  question: 
"What  existing  subjective  probability 
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assessment  technique  would  best  assess  the 
magnitude  of  uncertainty  in  a  given  weapon 
system's  development  effort?"  Grayson  and 
Lanclos  conducted  an  in-depth  review  of  the 
critical  literature  available  concerning 
techniques  and  methods  for  assessing  risk  and 
measuring  uncertainty  concerning  future 
events.  In  their  study,  the  researchers  con¬ 
sidered  the  statistical  and  psychological 
aspects  of  subjective  probabilities  and 
applied  content  analysis  to  six  techniques 
for  assessing  subjective  probabilities  which 
have  been  used  in  the  past. 

The  Choice-Between-Gambl es  is  a  technique 
that  employs  betting-type  situations  relative 
to  gambling  in  an  attempt  to  obtain  probabil¬ 
ity  and  cumulative  density  functions.  To 
accomplish  this,  experts  are  offered  choices 
between  a  "real-world"  gambling  situation 
which  includes  values  for  an  item  with 
unspecified  probabilities  and  a  hypothetical 
situation  about  two  outcomes  with  probabili¬ 
ties  specified.  A  series  of  iterations  that 
include  varying  probabilities  are  conducted 
to  derive  the  density  functions  desired. 

The  Standard  Lottery  is  a  technique  for 
deriving  a  probability  density  function  which 
covers  all  possible  values  of  a  given  item 
characteristic.  It  is  similar  to  the 
Cho ice-Between-Gambles  in  that  it  is  used  to 
present  an  expert  with  two  gambling 
situations.  The  technique  involves  selecting 
a  number  of  lottery  tickets  from  a  pool  of 
10U  that  is  varied  to  achieve  the  point  of 
indif ference.  The  expert  can  choose  as  many 
as  he  wants.  After  the  choice  is  made,  a 
ticket  is  drawn  at  random.  The  lottery  is 
used  as  a  standard  comparison  to  assist  the 
expert  to  decide  on  a  probability  value  for  a 
certain  outcome  which  relates  to  a  charac¬ 
teristic  of  performance  for  an  item. 

The  Churchman-Ackoff  technique  does  not 
use  a  gambling-type  situation  or  a  level  of 
indifference  concerning  characteristics  or 
outcomes.  The  expert  is  confronted  with 
evaluations  using  judgments  that  require 
ranking  of  relative  probabilities  between 
sets  of  values.  For  example,  "greater  than," 
"equal  to,"  or  "less  than"  choices  are  used. 
The  expert  must  decide  ranges  of  possible 
values  for  "real-world"  events  and  these  are 
then  converted  to  a  probability  density 
function  (b,  30-33). 

The  DLL  PH  I  technique  is  used  to  solicit 
group  consensus  about  possible  outcomes  or 
events  in  the  future.  Members  of  the  group 
of  experts  are  not  privy  to  the  other 
participants'  identities.  This  feature  mini¬ 
mizes  the  possibility  of  dominant  per¬ 
sonalities  and  group  pressure  in  obtaining 
final  results.  The  method  involves  rounds  of 


interrogation  which  include  controlled  feed¬ 
back  between  rounds.  Each  participant  can 
change  his  previous  choice  based  on  the  feed¬ 
back  from  each  round.  These  iterations  con¬ 
tinue  until  no  change  occurs.  The  results 
are  then  used  to  assess  and  form  a  measure  of 
group  consensus. 

The  DeGroot  Consensus  technique  also  uses 
experts  to  reach  a  group  consensus.  However, 
each  member  is  required  to  evaluate  a  proba¬ 
bility  for  an  unknown  value  of  a  parameter. 
Then  each  participant  is  exposed  to  the 
inputs  from  other  participants  and  revises 
his  own  judgment  after  assessing  the  others' 
importance,  expertise,  etc.  This  process 
continues  until  no  more  changes  occur  (5, 

39). 

Direct  Estimation  involves  an  estimating 
effort  that  gives  it  its  name;  each  expert 
appproximates  or  estimates  probabilities 
directly  with  no  exposure  to  other  probabili¬ 
ties  or  situations.  These  results  are  used 
directly  to  derive  a  probabilities  distribu¬ 
tion  for  future  events  or  outcomes  (5,  40). 

Research  Findings.  Grayson  and  Lanclos 
applied  content  analysis  to  each  technique 
studied  to  determine  which  technique(s) 
was(were)  commonly  used  for  measuring 
uncertainty.  The  researchers  found  that 
DELPHI  lias  been  widely  critiqued  in  the 
literature,  and  its  use  is  widespread  and 
popular. 

The  team  also  found  that  critical  analy¬ 
sis  and  testing  of  the  six  techniques  were 
fragmented  and  lacked  continuity  and 
consistency.  Based  on  these  findings, 

Grayson  and  Lanclos  recommended  that  a  series 
of  tests  be  conducted  to  determine  which 
techniques  or  combinations  thereof  apply  to 
measuring  uncertainty  for  weapon  systems 
development.  They  suggested  that  the  entro- 
pic  cost  model  could  be  used  to  measure  and 
compare  the  results  of  these  tests. 

Overall  Conclusions  from  Completed  Research: 

Obviously,  validity  testing  of  the  entro- 
pic  cost  model  is  not  complete.  The  Grayson 
and  Lanclos  effort  supports  DELPHI  as  a  popu¬ 
lar  and  widespread  tool  for  measuring 
uncertainty.  But,  popularity  and  widespread 
use  does  not  indicate  that  a  technique  is  the 
"one  best  way."  However,  the  research  does 
provide  some  implications  for  procurement  and 
acquisition  researchers  and  managers.  These 
implications  provide  a  logical  baseline  for 
early  identification  of  uncertainties  con¬ 
cerning  outcomes  for  weapon  system 
development.  Once  uncertainty  is  known  to 
exist  concerning  development,  management  can 
better  use  its  information  base  to  deal  with 
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risks  related  to  the  uncertainty. 
Development  of  such  a  capability  presents  a 
challenge  to  researchers  and  managers. 

IMPLICATIONS  AND  CHALLENGES 


Although  intensive  and  widespread,  re¬ 
search  has  not  progressed  to  the  point  of 
providing  management  with  an  effective  set 
of  tools  for  realistically  forecasting  the 
ultimate  cost  of  a  new  system.  Costing 
methodologies  vary  and  few  emphasize  the  for¬ 
mal  use  of  methods  for  capturing  and  measur¬ 
ing  technical  uncertainty.  The  research 
challenge  is  to  invent,  design,  and  innovate 
methods  for  quantifying  uncertainty  to  feed 
models  such  as  the  entropic  cost  model. 

These  efforts  will  be  difficult  at  best; 
there  will  be  failures  and  hopefully 
successes.  However,  this  series  of  research 
efforts  have  implications  for  management 
also.  The  qualitative  aspects  of  the 
research  may  be  useful  for  structuring  the 
assessment  evaluations  of  doubt,  reviewing 
status,  and  improving  planning  baselines. 

Each  of  the  techniques  for  assessing  and 
measuring  uncertainty  which  were  studied  by 
Grayson  and  Lanclos  depends  on  identification 
of  areas  or  factors  that  have  uncertain 
outcomes.  These  areas  should  be  solicited 
from  program  participants  early  in  the 
planning  cycle  prior  to  contract  award. 
Current  directives  specifically  address  mini¬ 
mizing  uncertainty  and  risk  (3,  6)  during  the 
acquisition  cycle.  Uncertain  factors  should 
be  identified  early.  Such  an  exercise  pro¬ 
vides  a  structured  approach  to  capturing 
the  judgment  of  managers,  evaluators,  and 
experts;  key  participants  usually  have  a 
knowledge  and  experience  base  that  they  draw 
on  to  "judge"  and  in  turn  identify  concern  or 
uncertainty  based  on  available  information. 
Management  could  do  well  to  capitalize  on 
this  knowledge  and  experience  by  soliciting 
participant  concerns  and  uncertainty  early  in 
the  pre-award  planning.  This  effort  should 
be  a  combined  exercise  by  both  the 
contracting  officer  and  program  manager  sup¬ 
ported  by  their  functional  staff.  One  major 
criticism  of  such  exercises  is  that  of  time— 
but  some  time  spent  early  may  have  some  long 
range  payoffs  in  terms  of  readiness  to 
respond  to  contingencies  during  development 
and  to  minimize  downstream  costs. 

Once  this  "uncertainty  baseline"  is 
established,  it  could  be  updated  at  major 
milestones  during  performance  of  the  develop¬ 
ment  contract.  Such  an  approach  could  also 
benefit  the  staff  in  preparing  for  major 
system  milestones  required  by  addressing  such 
factors  in  the  Decision  Coordinating  Paper 
(4). 


The  baseline  should  be  updated 
immediately  after  development  contract  award 
to  capture  specific  information  relevant  to 
the  particular  contractor( s)  involved  and 
evaluator's  beliefs  in  their  capability  to 
perform  the  contract. 

The  thrust  of  the  implications  are  not 
really  new.  But  the  main  purpose  of  this 
discussion  is  to  prompt  management  to  use 
their  talent  and  ability  to  establish  such  a 
baseline  for  dealing  with  uncertainty.  The 
ultimate  benefits  should  be  obvious:  better 
visibility  and  understanding  of  uncertainty 
assessment  problems  which  are  encountered  by 
contract  and  program  managers.  A  structured 
"audit  trail"  should  help  managers  to  get  at 
the  "root  cause"  of  some  of  our  problems  and 
treat  them  accordingly. 

If  a  problem  is  identified  which  cannot 
be  treated  readily,  at  least  the  researcher 
can  shed  some  light  on  an  approach  to  help 
better  address  the  problems  and  issues  at 
hand.  This  approach  takes  time,  but  the 
potential  payoffs  may  greatly  exceed  the 
investment. 
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ABSTRACT 


A  new  form  of  uncertainty  called  possibilistic 
uncertainty  is  introduced.  As  opposed  to  prob- 
alistic  uncertainty,  which  is  based  upon  an 
additive  measure  and  is  applicable  in  cases  of. 
repeated  experiments,  possibilistic  uncertain¬ 
ty  is  based  upon  a  non-additive  measure  and  is 
a  generalization  of  the  idea  of  ease  of  attain-* 
ment  in  a  situation.  We  discuss  the  properties 
of  possibilistic  uncertainty  and  describe  some 
prototypical  examples.  We  discuss  the  idea  of 
language  as  being  a  generator  of  possibilistic 
variables.  We  introduce  fuzzy  subsets  as  a 
means  of  translating  linguistic  values  into 
possibility  distributions.  We  discuss  the  idea' 
of  approximate  reasoning  as  a  means  of  simula¬ 
ting  a  large  class  of  human  reasoning  opera¬ 
tions.  We  introduce  a  measure  of  specificity 
of  a  possibility  distribution.  We  discuss 
applications  of  fuzzy  set  theory  to  intelligent 
quering  of  data  bases  and  multiple  objective 
decision  making.  Finally,  we  introduce  some 
ideas  from  fuzzy  arithmetic. 


INTRODUCTION 


There  appears  to  exist,  in  addition  to  the 
concept  of  probability,  another  idea  of  uncer¬ 
tainty.  This  concept  of  uncertainty  is  a  gen¬ 
eralization  of  the  idea  of  ease  of  attainment 
with  which  a  variable  can  assume  a  value.  This 
new  concept  of  uncertainty  has  been  called 
possibilistic  uncertainty  (1). 

Consider  an  optimization  problem  in  which  the 
set  of  possible  alternatives  consist  of  a  set 
X.  Assume  we  have  some  objective  function 
which  we  are  trying  to  optimize.  We  can  ob¬ 
serve  that  we  are  uncertain  as  to  which  element 
x6X  is  the  optimal.  Hence,  there  exists  some 
uncertainty  as  to  the  optimal  value,  however, 
this  uncertainty  is  not  of  a  probabilistic 
nature.  As  we  investigate  the  objective  func¬ 
tion  we  may  find  some  information  about  the 
optimal  value.  This  process  reduces  the  uncer¬ 
tainty  by  reducing  the  possible  values  which 
the  optimal  can  assume. 

We  can  consider  a  house  which  has  a  number  of 
windows  and  we  are  interested  in  the  ease  of 
burgl ari zati on  of  this  house.  It  appears  that 
the  concept  which  most  closely  approximates 


'our  analysis  of  the  situation  is  the  possibil¬ 
ity  of  entry. 

|A  quantitative  framework  has  been  suggested  by 
L.A.  Zadeh  (1)  Into  order  to  investigate  and 
apply  the  concept  of  possibility. 

Assume  V  is  a  variable  which  can  assume  values 
in  the  set  X.  A  possibility  distribution  TT  is 
mapping  from  X  into  [0,1], 

TT:  X - -[0,1] 

such  that  for  each  x6X,TT(x)  indicates  the 
possibility  of  V  assuming  the  value  x.  It 
.should  be  noted  that  the  largerTT(x)  the  more 
possible  the  circumstance  that  V  =  x.  Further¬ 
more,  if  A  is  a  subset  of  X  then  the 

Poss  [V6A]  -  Max  TT(x) . 

xGA 

In  order  to  understand  the  rational  behind  the 
selection  of  this  operation,  we  shall  use  our 
burglar  example  as  a  framework.  If  X  is  the 
set  of  all  the  windows  in  our  house  than  Tl(x), 
for  each  x€X  indicates  the  ease  or  possibility 
of  getting  in  window  x.  Let  A  be  some  subset 
of  windows,  AGX,  we  may  ask  how  easy  it  is  for 
a  burglar  to  enter  the  house  through  any  window 
in  this  subset.  The  answer  is  that  the  ease  of 
entering  the  house  through  a  window  in  the  sub¬ 
set  A  is  equal  to  the  possibility  or  ease  of 
entering  the  most  accessible  window  in  A.  Thu  - , 
the  possibility  of  entry  through  any  window  in 
A  is  equal  MaxTT(x)  for  x6A. 

Thus,  while  both  possibilistic  and  probabilistic 
uncertainty  are  mappings  from  a  set  into  the 
.unit  interval  they  have  different  properties. 
First  we  note  that  the  possibility  associated 
with  a  subset  is  the  maximum  possibility  of 
the  elements  in  that  subset,  whereas  the  prob¬ 
ability  associated  with  a  subset  is  the  sum  of 
the  probabilities  associated  with  the  elements 
in  the  set.  A  second  distinction  is  that  the 
sum  of  the  probabilities  must  sum  to  one  which 
is  not  the  case  for  possibilities. 

It  should  be  also  noted  that  probability  and 
possibility  are  addressing  two  different  ques¬ 
tions.  When  using  a  probabilistic  type  model, 
we  are  answering  a  question  about  what  percen¬ 
tage  of  the  number  of  times  we  perform  an  ex¬ 
periment  will  a  given  outcome  occur.  Whereas 
with  possibility,  we  are  addressing  questions 
1  about  how  easy  it  is  for  a  particular  outcome 
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to  occur  in  a  given  time. 


LINGUISTIC  VARIABLES 


In  many  instances  the  information  with  resoect 
to  the  possibility  distribution  associated  with> 
a  variable  can  be  inferred  from  information 
conveyed  via  natural  language.  We  shall  dis¬ 
cuss  the  connection  between  natural  language 
and  possibility  theory  based  upon  the  conceots 
of  linguistic  variables  and  fuzzy  subsets. 

The  idea  of  a  fuzzy  set  was  first  suggested  by 
L.A.  Zadeh  (2),  A  complete  bibliography  on 
fuzzy  sets  and  related  topics  can  be  found  in 
(3). 

Assume  X  is  a  set  of  elements  a  fuzzy  subset  A 
of  X  associates  with  each  element  a  grade  of 
membership  A(x)  contained  in  the  unit  interval. 
A  distinction  between  fuzzy  subsets  and  classic 
or  crisp  subsets  is  that  the  membership  grades 
for  fuzzy  subsets  are  in  the  unit  interval 
whereas,  crisp  subsets  are  in  essence  a  soecial 
case  of  fuzzy  subsets.  Fuzzy  subsets  provide 
us  with  a  mechanism  for  describing  concepts 
which  have  some  imprecision  in  thei r  definition. . 
For  example,  if  X  is  a  set  of  peoole  and  A  is 
the  subset  of  "tall"  oeople,  it  would  seem  more 
natural  to  use  a  fuzzy  subset  to  describe  A 
than  a  crisp  set. 

Extensions  of  the  usual  set  theoretic  operations 
to  fuzzy  subsets  have  been  suggested  (2). 


i is  a  subset  of  X  x  Y  such  that  the  pair  (x,y) 
;is  contained  in  S  if  they  are  related  by  the 
property  defining  S.  In  many  instances  the 
idea  defining  the  relationship  is  imprecise. 

For  example,  if  X  =  Y  =  Reals  and  if  S  =  '‘close 
to"  it  would  appear  that  we  could  best  describe 
S  by  a  fuzzy  rel ationshi o.  A  fuzzy  relation¬ 
ship  is  a  fuzzy  subset  of  X  x  Y. 

From  the  above  discussion  we  can  see  that  many 
concepts  are  best  defined  by  fuzzy  subsets, 
particul arly  those  used  in  natural  language. 

(Assume  V  is  a  variable,  for  example,  let  V  be 
the  speed  of  a  car.  In  many  instances,  parti¬ 
cularly  in  human  discourse,  the  value  assumed 
by  V  is  an  imprecise  value,  i.e.,  "about  75 
•  mi  1 es  per  hour"  or  "very  fast"  or  "slow."  With 
the  aid  of  fuzzy  set  theory,  we  can  represent 
this  type  of  information  in  a  quantitative 
fashion.  Let  X  be  the  set  of  values  which  the 
{variable  V  can  assume.  In  our  example,  it 
would  be  {0,100(,  the  set  of  speeds  the  car  can 
assume.  If  our  car  is  going  75  miles/hour,  we 
would  say 

V  =  75. 

If  the  car  is  going  "about  75,"  we  could  say 
V  =  "about  75," 

V  then  becomes  a  linguistic  variable. 

Furthermore,  the  concent  "about  75"  is  an  im¬ 
precise  concept  which  could  be  reoresented  as  a 
fuzzy  subset  A  of  X.  Thus,  we  can  say 


Assume  A  and  B  are  two  fuzzy  subsets  of  X  and 
let  x  be  a  typical  element  in  X. 

The  union  C  =  AUB  is  also  a  fuzzy  subset  of 
X,  where 

C(x)  =  Max  [A(x) ,  B ( x ) ] . 

The  intersection  D  =  An B  is  also  a  fuzzy  sub- 
set  of  X,  where 

D(x)  =  Min  [A(x) ,  B(x)] .  | 

The  negation  A'  is  also  a  fuzzy  subset  of  X, 
where  ' 


It  can  be  easily  shown  that  these  operations  »  i 
collapse  to  the  usual  set  theoretic  ooerations  j 
when  the  membership  grades  are  drawn  from  ; 

! 

It  should  also  be  noted  that  while  these  ooera¬ 
tions  are  the  most  often  used  they  are  not  nec-  ; 
essarily  the  only  possible  definitions  (4,5).  i 

We  can  also  generalize  the  concept  of  a  rel ati on-  ! 
ship  to  include  fuzzy  relationships.  Recalling* 
that  if  X  and  Y  are  two  sets,  aj^elationship  S  « 


V  =  A, 


where  A  is  a  fuzzy  subset  of  X. 


Thus,  the  value  of  a  linguistic  variable  can  be! 
equated  to  the  fuzzy  subset  representing  the 
linguistic  value,  in  this  case  "about  75." 


■We  note  that  the  statement 


V  =  "about  75" 


tells  us  something  about  the  value  of  V  but 
there  exists  some  uncertainty  as  to  its  exact 
value.  This  tvoe  of  uncertainty  with  respect 
,to  our  knowledge  of  V  is  what  we  have  oreviously 
called  possibil istic  uncertainty.  It  has  been 
suggested  that  the  possibility  distribution 
associated  with  the  variable  V  can  be  obtained 
ifrom  the  fuzzy  subset  A  representing  the  value 
bf-V. 

i 


jthus,  if  V  is  a  variable  with  base  set  X  such  , 
jthat  V  =  A,  where  A  is  a  fuzzy  subset  of  X,  theni 
A  induces  a  possibility  distribution  over  the 
|set  X  s.t.  for  each  x6X 


l 


TT(x)  =  A(x) , 


„  I 
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where  A(x)  is  the  membership  grade  of  x  in  A. 

In  order  to  see  the  validity  of  this  relation¬ 
ship  we  consider  the  following.  Assume  B  is  an 
ordinary  set  we  can  define  POSS  (B),  as  the 
possibility  of  finding  an  element  in  B.  Thus, 

POSS  (B)  =  Max  B(x) . 
x€X 

If  B  is  a  fuzzy  subset  of  X  the  definition  also 
holds.  Furthermore,  assume  that  A  is  also  a 
fuzzy  subset  of  X  than  the  possibility  of  ob¬ 
taining  an  element  in  A  OB  is 

POSS  [A O  B]  =  Max  [C(x)] 
xS 

where  C  =  AOB  and  hence 

C(x)  =  A(x)a  B(x).  (a  =  min) 

Let  us  consider  the  special  case  when  A  =  xi. 
Then  C  is  such  that 


3)  The  question  of  what  possible  values  can 
we  infer  for  the  variables  based  upon  the 
information  contained  in  one  and  two. 

In  the  followinq  we  shall  describe  a  lanquage 
which  will  enable  a  decision  maker  to  computer¬ 
ize  complex  approximate  reasoninq  problems. 

This  lanquaae  based  upon  the  theory  or  possi¬ 
bility  and  fuzzy  sets  is  called  PRUF  (6). 

PRUF  consists  of  two  aspects  translation  rules 
and  inference  rules.  The  translation  rules 
allow  us  to  convert  our  data  and  our  statements 
of  relationships  between  variables  into  possi¬ 
bility  distributions.  Our  rules  of  inference 
allows  us  to  make  inferences,  in  terms  of  possi¬ 
bility  distributions,  about  the  variables 
desired. 

Assume  V  is  a  variable  which  can  assume  values 
from  the  set  X.  The  statement  V  =  F,  where  F 
is  a  linguistic  value  expressed  as  a  fuzzy 
subset  F  of  X  translate  into  a  possibility 
distribution  riy  on  X  such  that 


C(x)  =  0  xi  *  x  nv(x)  =  F(x) 

C(x)  -  1a  B  (xi)  =  B(xi)  x  =  xi.  for  each  The  statement 


This  is  then  the  possibility  of  xi,  which  is 
denoted  n(xi),  hence 

n(xi)  =  Max  C(x)  =  B(x ) . 
x6a 


APPROXIMATE  REASONING 


Much  of  the  reasoning  performed  by  human  beings 
involves  a  type  of  reasoning  which  we  shall 
call  approximate  reasoning.  By  approximate 
reasoning,  we  shall  mean  a  type  of  logical 
inference  in  which  the  information  used  may  be 
imprecise. 


V  =  mF 

where  m  is  a  modifier,  such  as  not,  very,  sort 
of,  etc.,  translates  into  a  possibility  distri¬ 
bution  nv  on  X  such  that 

nv(x)  =  f+(x  ) 

where  F+  is  the  modification  of  F  induced  by  m. 
For  example,  if  m  =  not  than  F+(x)  =  1  -  F(x), 
if  m  =  very,  than  F+fx)  =  F2 ( x )  or  if  m  =  sort 
of,  than  F+(x)  =  Fl/1(x). 

As  a  simple  illustration,  consider  the  variable 
V  where 

X  =  {0.1,2. 3,4,5, 6f. 


An  example  of  approximate  reasoning  would  be 
the  following: 

"If  x  is  near  50,  then  y  is  much 
greater  than  100" 

x  =  about  45 
what  is  Y? 


Consider  the  statement  V  =  small,  where 

small  =  H,  U  jj3 ,  ^_5,  J_,  0,  01 
10  1  2  3  4  5  6  f  * 

Then  not  small  is 

/0,  0,  .2,  .5,  .7,  1,  11 
Iff  T  T  T  T  5  6/ 


In  a  structural  sense  approximate  reasoning 
consists  of  the  following: 


and.  very  small  induces  the  possibility  distri¬ 
bution 


1)  A  set  of  statements  describing  the  relation¬ 
ships  between  variables,  i.e.,  "if  x  is 
near  50  then  y  is  much  greater  than  100." 
These  rules  in  general  involve  imprecise 
values  for  the  variables. 


I’  I’  jii.  A >  j_09,  0,  0  1 
0  1  2  3  4  5  6  /. 

Assume  again  V  is  a  variable  taking  values  in 
the  set  X  and  U  is  another  variable  takinq 
values  in  the  set  Y.  Let  F  and  G  be  fuzzy  sub 
sets  of  X  and  Y  representing  the  linguistic 


2)  Imprecise  data  on  values  for  some  of  the 
variables,  i.e.,  x  =  about  45. 
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values  associated  with  V  and  U  respectively. 
The  statement 


My)  =  Max  [Min(Ilv(x),  nu/v(x,y)l. 

vPT  J 


V  is  F  and  U  is  G 

induces  a  possibility  distribution  on  the 
binary  variable  (V,U)  defined  over  the  set 
X  x  Y  such  that 

nv,u(x>y)  =  Min  [f(x)»  G(y)] 

for  each  (x,y)  8  XxY. 

As  a  very  simple  illustration  consider 


II 

X 

Y  = 

11. 

c  - 

/l. 

.6, 

r  — 

\l 

2 

G  = 

/o. 

.3, 

h 

2 

As  a  simple  illustration  of  the  above  procedure, 
we  can  consider  the  following. 

Let  U  and  V  be  variable0  measured  over  the  sets 
Y  =  {-1,2,3}  and  X  =  ja,b[,  let  G  and  F  be  fuzzy 
subsets  of  Y  and  X 


1,  .6,  .1 

T  T  T 


JL 

a  b  j 


} 


Assume  we  have  the  following  information 


P'  :  if  V  *  sort  of  F  then  U  =  G 
P"  :  if  V  *  very  F  then  U  =  not  G 
V  =  F. 

First  we  can  see  that 


then  the  statement 

"V  =  F  and  U  =  G," 

induces  the  joint  possibility  distribution 


.6  0  ,  .1  ,  .1  \ 
TO?  TOT  TOT  TOT  / . 


sort  of  F  =  F1/2  j-—*  -liij. 

very  F  -  F2  =  -f1) 

Not  G  =  1  -  G  =  JO,  .4,  .9) 

IT  T  T/ 

Using  our  translation  rules,  we  obtain  that  P 
translates  into 


The  statement 


V  is  F  or  U  is  G 

induces  a  possibility  distribution  on  the 
binary  variable  (V,U)  defined  on  the  set  XxY 
s.  t. 

nV)U(x>y)  =  Max  [f(x)>  s(y)]. 

The  statement 

if  B  is  F  then  U  is  G 

induces  a  possibility  distribution  on  the  bi- 
nary  variable  (V,U)  defined  on  the  set  X  x  Y 
s.t. 


nu/v(x.y)  =  Min  £l,l-G(x)  +  F (y )J  . 

A  fundamental  rule  of  inference  in  approximate 
reasoning  is  the  rule  of  fuzzy  compositional 
inference. 


and  p"  translates  into 


Since  the  logical  connection  between  our  two 
statements  is  that  P'  and  P"  then  together  they 
generate  an  overall  possibility  distribution. 


P’  and  PH  Ou/v  where 
nu/v(x,y)  “  Min  [riy/y(x,y),  riy/7(x,y)J 


thus  we  get 


Then  using  the  compositional  rule  of  Inference 
we  get: 


Assume  nu/v  is  a  possibility  distribution  over 
the  binary  variable  (V,U).  Let  JIV  be  a  possi¬ 
bility  distribution  representing  the  value  of 
the  variable  V,  then  from  this,  using  the  rule 
of  fuzzy  compositional  inference  we  can  infer 
the  possibility  distribution  associated  with 
the  variable  "u"  as 


flu  =  i  .9,  .4,  A) 

IT  2  3  / 

as  the  possibility  distribution  associated  with 
the  variabl  e  U. 


188 


SPECIFICITY  OF  POSSIBILITY  DISTRIBUTIONS 


INTELLIGENT  QUERING  OF  DATA  BASES 


T* 


Assume  a  variable  V  has  an  associated  possibi¬ 
lity  distribution  F  defined  over  the  finite 
base  set  X. 


An  a  level  set  associated  with  F  is  the  crisp 
subset  of  X  denoted  Fa  defined  by 

Fa  =  |x/F(x)  >  a,  xGxj. 

Fa  then  is  the  set  of  all  elements  have  possi- 
bil  i ty  at  1  east  a. 


Using  this  concept  of  level  sets  we  can  intro¬ 
duce  the  idea  of  the  specificity  of  a  possibi- 
1 ity  distribution  as 


S(F,X) 


/-a  max 

1  c 


1 


Card  F( 


a 


da. 


Fuzzy  sets  provide  a  methodology  for  intelli¬ 
gently  quering  a  data  base  via  applying  certain 
rules  for  interpreting  quantified  statements. 
(8,9) 

Assume  we  have  a  data  base  consisting  of  the 
heights  of  the  people  in  a  set  X.  Assume  we 
ask  the  question,  “How  true  is  it  that  most 
people  in  the  set  are  tall?''  The  question 
Involves  two  concepts  which  need  definition: 
tall  and  most.  We  must  first  ask  the  question¬ 
er  what  he  means  by  tall.  Since  tall  is  an 
Imprecise  concept,  he  can  use  a  fuzzy  subset 
to  represent  the  concept  of  tall .  Let  us 
assume  that  this  is  supplied  via  the  fuzzy 
subset  F 


F:  Y  — ► I 


Card  Fa  is  the  number  of  elements  in  Fa  and 
a  max  is  the  maximum  possibility  associated 
with  any  element  in  X.  (7) 


where  Y  is  the  set  of  heights  30  to  80  inches. 
Since  each  x6X  has  a  height  belonging  to  Y,  we 
can  construct  a  fuzzy  subset. 


Exampl e: 


F*:  X  — ►  I 


Assume  F  =  f 

then 


a  <  .3 

Fa  ■ 

)a,b,c 

.3<a<  .6 

pH 

II 

jb,c} 

a  <  .  6 

pH 

II 

lcl 

S(F.X)  -  f 

1  1 

— —  da  = 

K 

card 

Fa 

card  Fa  =  3 
card  Fa  =  2 
card  Fa  =  1 

+  A  +  A  =  *65 
2  1 


Where  if  yi  is  the  height  of  xi6X  then  F*(xi)  = 

F(yi ) . 

Next,  we  must  inquire  as  to  the  meaning  of  the 
word  "most."  Again,  this  being  an  imprecise 
concept,  it  can  be  described  by  a  fuzzy  subset, 

M:  I  — ►  I, 

where  for  each  U0I ,  M(u)  indicates  the  degree 
to  which  u  portion  of  the  people  being  tall 
satisfies  our  concept  most. 


The  measure  of  specificity,  which  always  lies 
in  the  unit  interval,  measures  the  degree  to 
which  the  possibility  distribution  points  to 
one  and  only  one  of  the  elements  of  X  as  its 
manifestation.  The  speci fici ty  measure  in 
possibility  theory  is  analogous  to  the  con¬ 
cept  of  entropy  in  probability  theory. 

The  measure  of  spcificity  plays  an  important 
role  in  evaluation  of  the  performance  of  fore¬ 
casters.  Assume  we  have  a  forecaster  whose 
forecasts  are  in  terms  of  linguistic  variables 
representable  as  possi bil  i Stic  distributions. 
Let  F  be  a  fuzzy  subset  of  X  representing  a 
forecast.  A  good  forecast  should  have  two 
characteri stics--i t  should  be  specific  there¬ 
by  giving  us  the  most  information  and  it  should 
be  correct.  Assume  that  the  true  value  of  the 
forecasted  variable  is  x6X,  then  the  degree 
of  truth  of  the  forecast  is  F(x).  We  can 
measure  the  specificity  of  our  forecast  as 
$(F,X).  The  product  of  these  two  terms  mea¬ 
sures  the  performance  of  the  forecaster.  By 
calculating  the  average  of  these  products  over 
a  sampling  of  situations,  we  can  obtain  a  mean 
measure  of  performance  of  a  forecasting  system. 


Finally,  using  the  translation  rule  for  quanti¬ 
fied  statements  supplied  by  PRUF,  we  can  express 
the  truth  of  the  question  as: 

i*  i 

where  N  is  the  number  of  elements  in  X. 


MULTIPLE  OBJECTIVE  DECISION  MAKING 

Fuzzy  set  theory  provides  a  framework  in  which 
to  construct  models  for  multiple  objective 
decision  making  (10). 

Assume  we  have  a  decision  in  which  we  have  a 
set  X  of  alternatives  and  a  class  A^  ,  A2 , . .  .Ap 
of  objectives.  We  can  represent  each  objective 
as  a  fuzzy  subset  of  X,  where  Ai(xj)  indicates 
the  degree  to  which  xj  satisfies  the  ith  objec¬ 
tive. 

Next  we  must  consider  how  our  objectives  should 
be  combined  to  formulate  our  decision  function  D. 


189 


» 


This  page  not  submitted. 


t, 


190 


If  our  problem  requires  the  satisfaction  of  A 
and  Ai  and  A2...and  An,  then  we  can  formulate 
D  as 

D  =  AiO  A2OA3...nAn 

D  in  this  case  becomes  a  fuzzy  set  of  X  and  we 
select  as  our  optimal  alternative  the  element 
which  the  largest  grade  of  membership  tn  D. 

More  complex  decision  criteria  can  be  modeled 
using  the  operations  of  fuzzy  sets. 

One  such  case  was  suggested  by  Yager  (11)  to 
handle  situations  in  which  the  objectives  have 
differing  degrees  of  importance.  He  suggested 
that  we  associate  with  each  objective  a  number 
in  the  unit  interval  indicating  its  importance 
and  then  raise  each  fuzzy  subset  corresponding 
to  an  obejctive  to  the  power  equal  to  its  im¬ 
portance. 

Thus , 

D  =  A“1OA“2<'VA“3n....A*"  , 

where  ai  is  the  importance  of  the  ith  objective 
and  A.1  is  defined  as  the  fuzzy  subset  of  X 
where 

A?‘  ( x )  =  (A1(x))“‘ 


FUZZY  ARITHMETIC 


The  division  of  one  fuzzy  number  by  a  second, 

H  =  F/G,  is  also  a  fuzzy  number  such  that 

u  -  F(x)  A  G( y) 

H  "  x/y  '  for  all  x,  yGR. 

A  methodology  for  comparing  fuzzy  numbers  has 
been  suggested  by  Yager  (14). 

Assume  F  is  a  fuzzy  number  with  maximal  grade 
of  membership  one.  Let  Fa  be  the  a-  level  set 
associated  with  F.  Let  Ma  be  the  mean  of  the 
elements  contained  in  Fa.  Then 

V(F)  =  [  1  Ma  da. 


is. a  mapping  from  the  set  of  fuzzy  subsets  of 
R.into  R.  We  can  use  V(F)  to  compare  the  mag¬ 
nitudes  of  fuzzy  numbers. 


CONCLUDING  REMARKS 


In  the  preceding  we  have  sketched  very  briefly 
some  of  the  areas  of  applications  of  fuzzy  set 
theory  and  the  related  idea  of  possibility 
theory.  We  feel  that  this  new  tool  provides  a 
fertile  framework  in  which  many  of  the  problems 
facing  our  complex  society  and  many  of  the 
opportunities  presented  by  the  new  technologies 
can  be  best  handled. 


A  fuzzy  number  is  a  fuzzy  subset  of  the  set  R 
of  reals.  Thus,  if  F  is  a  fuzzy  number  then: 

F:  R-[o,l]  (12,13) 


In  the  following  we  shall  assume  F  and  G  are 
fuzzy  numbers.  The  sum  of  two  fuzzy  numbers, 
F  +  G  =*  H  is  also  a  fuzzy  number  such  that 


H  = 


for  all  x,  ySR 


•  F  s  < 

L2. 

,  -5, 

.7 

,  1. 

.6 

,  .1) 

•  1 

13 

4 

5 

6 

7 

8  ) 

G  -  < 

(.6, 

,  -1, 

.5 

\ 

l-l 

0 

1 

) 

=  F  +  G 

H 

1.2, 

12 

.5, 

3 

.6, 

4 

.6, 

T 

,  .1. 
6 

.5, 

3 

.7, 

4 

1,  . 

6, 

.1, 

.2, 

.5, 

.5, 

.5, 

.5,  . 

6  ; 

r 

8 

4 

T 

6 

7 

8 

H  =  \ 

1.2, 

12 

,  -5, 
3 

.6 

4 

,  .7 
5 

,  1, 
6 

,  .6, 
7 

.5, 

8 

i) 

The  difference  of  two  fuzzy  numbers,  F  -  G  -  H 
is  also  a  fuzzy  number  such  that 


for  all  x,  y6R  . 
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ABSTRACT 


This  is  a  brief  description  of  GAC’s  method¬ 
ology  to  assess  and  reduce  risk  during  the 
preliminary  product  design  stage.  Fundamental 
to  this  approach  is  the  expression  of  program 
element  costs  in  terms  of  a  range  of  probable 
costs,  rather  than  as  a  single  cost  associated 
with  a  point  design.  These  element  costs  are 
combined  to  a  program  cost  expressed  as  a 
probability  function,  rather  than  one  unique 
number.  This  not  only  provides  a  clearer 
picture  of  what  is  possible,  but  the  procedure 
itself  contains  the  means  to  identify  the 
principal  causes  of  cost  uncertainty. 

For  this  methodology  we  assume: 

•  a  system  made  up  of  a  set  of  elements  or 
subsystems 

•  total  system  cost  is  the  sum  of  the  element 
cos  ts 

•  a  Cost  Estimating  Relationship  (CER)  for 
each  element 

•  the  inputs  to  the  CER  are  treated  as  random 
variables 

•  the  element  costs  are  stochastically 
independent . 

We  have  a  set  of  inputs  with  assumed  density 
functions,  a  corresponding  set  of  CERs  and  a 
set  of  outputs,  which  are  summed  to  arrive  at 
the  total  cost  probability.  We  wish  to  esti¬ 
mate  for  each  output,  and  for  the  total,  a 
confidence  limit  (i.e.,  state  that  the  proba¬ 
bility  is  90%  the  cost  will  lie  between  two 
stated  limits) ,  and  for  any  arbitrary  cost, 
estimate  the  probability  that  that  cost  will 
be  the  arbitrary  value  or  less.  In  the  usual 
case,  we  use  the  first  four  additive  moments 
of  each  output  to  determine  our  confidence 
limits  and  probability  statements,  then  add 
these  moments  to  determine  the  cost  probabil¬ 
ity  characteristics  for  the  total  project. 

To  determine  the  output  characteristics,  ve 
may  use  either  a  transformation  of  variables 
and  determine  the  output  density  function,  or 

The  contents  of  this  paper  represent  the  views 
of  the  authors  and  are  not  necessarily  those 
of  the  Grumman  Aerospace  Corporation. 


use  an  analytical  approximation  involving  a 
Taylor  expansion  of  the  CER.  The  end  result 
is  an  analytical  approach  that  is  relatively 
simple,  economical,  and  provides  excellent 
viaibility  at  the  subsystem  or  element  level. 


INTRODUCTION 


Defining  program  costs  in  terms  of  a  probabil¬ 
ity  function,  rather  than  a  point  estimate, 
has  been  discussed  for  a  long  time.  Sobel 
(21)  and  Dienemann  (3)  proposed  the  use  of 
Monte  Carlo  in  military  cost  estimating  in  the 
early  1960s. 

By  1973,  FisherTs  paper  (9)  on  uncertainty, 
along  with  definitions  of  risk  and  various 
forms  of  uncertainty,  listed  seven  basic  types 
of  analysis  that  can  be  used  to  deal  with 
uncertainty.  They  included  Monte  Carlo, 
sensitivity  analysis,  ’’range  of  estimates”, 
adjustment  factors  (allowances),  supplemental 
(additional)  discounting,  afortiori  (worst 
case)  analysis  and  special  studies.  This 
variety  and  his  mentioning  that  he  would  only 
comment  briefly  on  each  method,  as  they  should 
be  familiar  to  most,  implies  a  healthy  growth 
process.  Klein  (14),  (15)  popularized  the  use 
of  the  Beta  distribution  to  represent  cost 
estimates,  which  is  probably  the  most  used 
technique  today,  especially  when  combined  with 
a  Monte  Carlo  summing  procedure. 

McNichols  PhD  Dissertation  (18)  in  1976  dis¬ 
cussed  an  analytical  alternative  to  Monte 
Carlo  using  the  moments  of  probability  func¬ 
tion.  This  paper  discusses  a  procedure  par¬ 
tially  derived  from  his  work.  It  has  evolved 
over  the  last  ten  years  and  is  presently  used 
internally  in  selected  cases  of  probable  high 
risk. 

Though  these  ideas  have  slowly  gathered 
strength  through  the  years  they  have  only 
recently  begun  to  acquire  official  recogni¬ 
tion.  The  recent  revision  to  DOD  Directive 
5000.2  states  that  cost  estimates  shall  be 
supplied  with  a  confidence  band  not  just  as  a 
point  estimate  using  best  available  means. 
However,  to  the  authors’  knowledge,  no  service 
has  yet  formally  implemented  this  policy. 
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METHODOLOGY  OVERVIEW 

Risk  is  defined  as  the  probability  that  a 
project  will  not  be  completed  within  specified 
cost,  time  and  performance  constraints  by 
following  a  specified  course  of  action  (16). 
Only  cost  risk  is  discussed  here.  However,  we 
often  use  cost  risk  as  a  proxy  for  performance 
risk.  We  have  found  that  it  is  easier  for  a 
designer  to  express  uncertainty  in  a  parameter 
by  an  estimate  of  its  range  rather  than  to 
quantify  probabilities  of  expected  perfor¬ 
mance. 


Further,  we  have  assumed  that  the  cost  elements 
and  CER  parameters  are  independent.  This  is 
not  necessary  in  theory,  but  dependence  creates 
two  problems.  First,  it  is  more  difficult  to 
handle  computationally.  Second,  the  degree  of 
dependency  must  be  given  or  inferred.  We  have 
found  it  very  difficult  to  express  this  depen¬ 
dency  in  a  meaningful  way.  This  latter  problem 
is  basic  and  if  we  could  solve  it,  computation¬ 
al  algorithms  could  easily  be  developed. 

An  overall  block  diagram  of  the  procedure  is 
shown  as  Fig.  1.  The  major  points  to  be 
discussed  are; 


Our  results  are  usually  presented  in  the  more 
conventional  sense  of  implied  successfulness 
(1-risk).  We  feel  that  this  makes  the  results 
easier  to  explain  and  causes  no  mathematical 
inconsistencies . 

The  process  described  here  implies; 

•  a  system  made  up  of  a  set  of  elements  or 
subsystems  -  a  Work  Breakdown  Structure 
(WBS)  or  set  of  cost  categories 

•  a  Cost  Estimating  Relationship  (CER)  for 
each  element  -  may  be  complex  but  can  be  a 
simple  multiplier,  or  a  direct  estimate 

•  a  total  system  cost  found  by  summing  the 
costs  of  these  elements. 

To  quantify  the  results  in  a  probability 
sense,  the  following  basic  assumptions  are 
made ; 

•  some  or  all  of  the  inputs  can  be  treated  as 
random  variables 

•  there  is  a  greatest  lower  bound  for  the 
resources  required  for  the  task  to  be 
accomplished  with  a  probability  of  zero 

•  there  is  a  least  upper  bound  for  the  re¬ 
sources  required  for  the  task  to  be  accom¬ 
plished  with  a  probability  of  one. 


•  how  to  quantify  the  spread  in  the  input 
variables 

•  the  analytic  techniques  used  to  translate 
their  uncertainties  into  output  uncer¬ 
tainties;  a  range  of  program  costs 

•  how  to  present  the  output  for  review  and 
how  it  can  be  used  to  either  isolate  the 
major  cost  drivers  or  to  determine  probable 
program  costs. 

SELECTION  OF  INPUT  FORMS 


On  the  basis  of  our  experience,  we  feel  that 
the  estimation  of  the  spread  in  the  input 
variables  is  the  most  difficult,  but  most 
important,  aspect  of  the  problem.  Any  reason¬ 
able  analytical  technique  will  provide  accept¬ 
able  results  given  good  input  data.  No  analyt¬ 
ical  approach,  regardless  of  sophistication, 
can  make  up  for  bad  input  data.  Therefore, 
much  of  our  efforts  have  been  to  find  the 
simplest  procedure  that  can  express  the  de¬ 
signer’s  or  estimator’s  often  intuitive  feel 
for  the  validity  of  his  cost  estimate  in  terms 
usable  to  the  cost  analyst. 

Various  approaches  have  been  used  to  aid  the 
designer  to  translate  his  intuitive  feelings 
into  hard  numbers.  Some  approaches  have 


INPUTS 


OUTPUT 


•  PHYSICAL  OR  PERFORMANCE 
PARAMETERS  FOR  CERs 

•  DIRECT  COST  ESTIMATES 


•  THE  DISTRIBUTION  OF 
POSSIBLE  COSTS  AS  A 
CDF 


•  UNUSUALLY  TRIANGULAR 

Fig.  1  -  Overview  of  Approach 


* 
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selected  a  functional  form,  usually  Beta  (13), 
(14),  (15)  Gamma,  Normal  or  Weibul.  Then, 
having  a  generic  form  they  have  either  se¬ 
lected  from  among  various  representations  of 
this  form  (by  selecting  the  most  appropriate 
picture,  etc.)*  or  used  some  estimate  of  the 
data  to  derive  the  values  of  the  parameters. 

A  popular  method  using  data  has  been  to  select 
three  data  values,  high,  low  and  mode,  as  per 
PERT,  etc.  Others  have  attempted  to  fit 
functional  forms  non-parametrically  using 
percentiles  or  subjective  probability  rankings 
(6),  (7),  (8),  (12),  (20).  The  most  popular 
approach  in  weapon  system  acquisition  litera¬ 
ture  appears  to  be  to  select  a  representative 
Beta  given  an  estimate  of  the  range  of  vari¬ 
ability.  In  practice,  we  have  found  that 
picking  a  Beta  shape  can  be  a  difficult  and 
non-repea table  process. 

In  advance,  the  true  density  function  is 
unknown  (and  probably  unknowable).  However, 
we  can  use  some  common  sense  and  give  our 
unknown  function  some  characteristics.  A 
reasonable  set  of  characteristics  would  be: 

•  finite  ends 

•  not  necessarily  symmetric 

•  unimodal 

•  computationally  simple. 

The  triangular  distribution  has  these  prop¬ 
erties.  It  is  bounded.  It  incorporates 
skewness,  an  important  cost  consideration.  It 
can  be  described  by  the  location  of  its  peak 
and  two  end  points;  the  peak  corresponding  to 
the  most  likely  cost  and  the  extreme  values 
related  to  the  most  optimistic  and  pessimistic 
values.  A  similar  view  has  since  been  ex¬ 
pressed  in  (2).  Further,  it  appears  to  be  an 
easy  concept  to  visualize  and  accept  especial¬ 
ly  for  those  unfamiliar  with  statistics.  An 
additional  benefit  is  that  it  is  fairly  easy 
to  handle  in  computation  and  simulation. 

Figure  2  illustrates  our  requirements  using 
this  distribution. 


—THE  MOST  LIKELY  VALUE  •  m 
-THE  LOWEST  VALUE  THAT  THIS  PARAMETER 
CAN  HAVE  -8 

-THE  HIGHEST  VALUE  THAT  THIS  PARAMETER 
CAN  HAVE  -  h _ 

Fig.  2  -  Standard  Input  Data 

To  the  best  of  our  knowledge,  not  many  rigor¬ 
ous  analyses  have  been  performed  comparing  the 


error  resulting  from  using  this  function  to 
Beta's  or  others,  other  than  (17),  which 
concluded  that  for  PERT  networks,  the  triangle 
was  as  good  as  the  Beta.  Also  Ref  (18)  re¬ 
viewed  and  catalogued  PERT  and  PERT  deriva¬ 
tives,  as  well  as  several  other  functions,  and 
discussed  error  magnitudes.  In  those  cases 
where  we  have  been  able  to  compare  to  Beta 
variables,  the  results  have  been  similar. 

Sometimes,  even  less  is  known  about  the  vari¬ 
ables  shape.  Then  we  suggest  the  use  of  the 
uniform  distribution,  where  any  value  is 
equally  likely  within  the  range  (Fig.  3). 

However,  if  the  form  of  the  input  distribution 
is  known  and  the  parameters  can  be  estimated, 
then  the  analytical  approach  described  here 
can  still  be  used.  We  presently  have  computer 
codes  to  handle  Normal,  Log  Normal,  Beta, 
uniform  and  triangular  inputs  or  mixtures  of 
any  or  all  of  the  above. 


1 _ _ *  INPUT 

£  h  VALUE 

Fig.  3  -  Input  (All  Data  Equally  Likely) 
MATH  MODEL 


We  have  proposed  an  analytical  approach  to  the 
problem  of  transforming  the  input  to  a  final 
output,  rather  than  the  more  popular  Monte 
Carlo  (3),  (11),  (23).  It  uses  the  moments  of 
the  probability  distributions,  rather  than  the 
random  variations  and  is  more  accurate  while 
using  less  computer  time.  This  is  especially 
true  when  a  large  number  of  inputs  must  be 
included  and/or  the  estimating  equations  are 
very  complex.  Further,  It  can  be  used  with  a 
small  computer,  or  even  a  hand  calculator,  if 
the  problem  is  not  too  large. 

It  allows  all  Input  parameters  to  be  entered 
at  their  applicable  level  in  the  Work  Break¬ 
down  Structure  (WBS).  If  entered  directly, 
the  moments  are  summed  to  the  appropriate 
level.  If  entered  via  CERs,  the  moments  of 
the  transformed  variable  are  summed  at  the 
appropriate  level.  All  moments  are  summed  at 
the  final  output  level  and  converted  to  a  Beta 
function  representing  the  total  cost  distribu¬ 
tion.  If  desired,  a  non-parametric  method  is 
available  to  compute  the  final  cost  distribu¬ 
tion  using  a  Tchebycheff  Markoff  approximation 
and  the  moments.  The  mechanics  of  the  various 
processes  are  handled  in  a  set  of  modular  sub¬ 
routines  coded  for  several  different  computers. 
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We  represent  each  function  by  some  form  of  its 
first  four  moments.  We  realize  that  this  is 
not  a  complete  representation,  but  an  approxi¬ 
mation.  It  has  been  our  experience  that  using 
only  the  mean  and  variance  are  not  enough; 
some  representation  of  non-symmetry  and  shape 
must  be  included. 

This  procedure  was  first  developed  in  (18)  and 
expanded  in  (22),  The  following  is  a  brief 
expository  account  and  the  details  and  anal¬ 
ytic  derivations  are  to  be  found  in  the  refer¬ 
enced  works. 


Thus,  given  a  random  variable  X  with  known 
mean  and  central  moments,  the  A  moments  may  be 
found. 

In  the  case  of  a  product  of  independent  vari¬ 
ables  (21)  shows  that  the  origin  moments  of 
the  independent  variables  may  be  multiplied  to 
get  the  origin  moments  of  the  product,  then  A 
moments  determined  from  the  following  relation¬ 
ships:  Where  M^(X)  is  the  i  th  origin  moment, 

M^(X)  *  E(X*)  and,  as  before,  is  the  i  th 

additive  moment. 


Three  moment  forms  are  required;  central, 
origin  (multiplicative),  and  additive.  For 
briefness,  the  latter  two  are  called  ,fM"  and 
"A”  moments,  respectively,  in  the  remainder  of 

this  paper.  The  i^  central  moment  has  the 
standard  textbook  definition  of 

E  [(X-E(X))  1  ]  =  f  (X-E(X) )  1j(X)  dx  =  u*’- 

J  -  co 


M2  =  A2  + 

M3  =  A3  +  3A1  A2  +  Ax3 

Ml*  =  \  +  3A22  +  1 »A1  A3  +  6Ax2  A2  +  A1k 


It  is  shown  in  (10)  that  the  first  four  A 
moments  of  the  density  functions  of  a  random 
variable  X  are  as  shown  below,  and  that  the  A 
moments  of  the  density  function  of  the  sum  of 
independent  variables  is  the  sum  of  these  A 
moments . 


A  =  E  (X)  =  u 

A^  »  E  (X2)  -  u2  =  u^2' 

A3  =  E  (X3)  -  3  E  (X2)  u  +  2u3  = 

A^  =  E  (XU)  -  1*  E  (X3)  u  -  3  E2  (X2)  + 
12  E  (X2)  u2  -  Su1* 

where 


and 


A3  =  M3  -  3MX  M2  +  2Mj 3 

AU  =  Ml+  ~  3M2<?  -  UMi‘M3  +  12M1 


These  relationships  make  the  determination  of 
moments  for  such  expressions  as 


A.  =  i  th  additive  moment  =  i"^  A  moment 

l 

u  =  mean  (first  origin  moment) 
u'x;  =  i  th  central  moment 


C  *  xx  \x2  +  x3  +  .  ..) 

easy  since  A  moments  may  be  used  inside  the 
parentheses,  the  M  moments  found,  multiplied 
by  the  M  moments  of  X^,  then  A  moments  of  C 
found  (Fig.  4). 


It  is  seen  that  the  first  three  A  moments  are 
exactly  the  same  as  the  first  origin  tnoment 
and  second  and  third  central  moments. 

Now,  by  definition, 

U(U)  =  E  (X-u)U  =  E  (X1*)  -  1*  E  (X3)  u  + 

6  E  (x2)  U2  -  3U1* 

3u(2)  =  3  E2  (X2)  -  6  E  (X2)  u2  +  3U1* 


In  a  computer  routine,  we  would  sum  each  of 
the  four  moments  of  the  variables  X^,  X^, 

etc.,  and  convert  the  sum  to  M  moments  (ATOM 
subroutine).  These  would  be  multiplied  by  the 
appropriate  M  moments  of  X^,  and  the  products 

converted  to  A  moments  (MTOA  subroutine)  of 
the  cost  function.  Many  simple  costing  prob¬ 
lems,  such  as  summing  WBS  elements  with  multi¬ 
pliers  can  be  solved  in  this  manner.  For 
complex  CERs  more  complex  procedures  must  be 
used . 


.00 


3u 


(2  y 


=  E  (X4)  -  It  E  (X3)  u  - 

3  E2  (X2)  +  12  E  (X2  )u2-6u**=A^ 


In  many  instances,  it  is  possible  to  determine 
the  CER  output  density  function,  and  hence  the 
moments  directly  by  transformation  of  vari¬ 
ables.  X  is  a  random  variable  with  density 
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form 


function  f(x),  and  Y  =  h  (X),  where  y  *  h  (x) 
defines  a  one-one  transformation.  The  inverse 
of  y  =  h  (x)  is  designated  by  x  ■  w  (y),  and 
the  derivative  dx/dy  is  designated  by  w'(y). 
The  density  function  of  Y  =  h  (X)  is  given  by 

g  (y)  *  f  (w  (y))  |w*  (y ) | 

where 

|  w'  (y)  |  is  the  absolute  value  of  w1  (y) 

For  a  CER  of  the  form  y  =  a  +  b  xn,  the  above 
technique  applies,  since  the  mapping  of  x  into 
y  is  one-one,  and  the  inverse  is  differen¬ 
tiable.  A  practical  requirement  is  that  x 
have  a  density  function  such  that  f  (w  (y))  is 
readily  solved. 

In  particular,  setting  a  »  o,  b  a  1,  n  *  1  we 
have  y  *  x  and  inputting  the  values  of  x  into 
our  subroutine  we  output  the  A  and  M  moments. 
For  example,  for  the  triangular  distribution 
defined  as  follows: 


f  (x)  =  'h-i.Vi:  (x-1}  1-x-m 
f  (x)  =  (h_x)  m-x-h 


f  (x)  =  0  elsewhere 


Y  -  bXn*  Zra 

can  be  separated  into  bXn  and  lZm  transformed 
via  TRIMO  (our  subroutine  that  handles  tri¬ 
angular  inputs),  the  two  sets  of  M  moments 
found,  multiplied  together,  converted  to  A 
moments  and  then  to  the  CDF  of  the  cost  func¬ 
tion. 


v 


c-x1  (x2  +  x3  +  x4) 


f  A21  +  a31  +  a41  *  AT1 
a22  +  A32  +  A42  =  AT2 

A23  +  A33  +  A43  =  AT3 
A24  +  A34  +  A44  =  AT4 


TT 


Aj‘S 

V _ 


AT0M 

SUBROUTINE 


MT2#M12  =  M*2 
MT3  ’  M13  *  Mtt3 
MT4#M14*Mrr4 


Mt'S 

_ y 


for 


y 

s  (y) 
g  (y) 


+  bx 


n 


nb  (h-l)  (m-1) 

_ 


nb  (h-l)  (h-l) 
g  (y)  =  0  elsewhere 
where 

m  =  modal  value  of  x 


(x-l)'L  l<x<m 


(h-x)^  n  m<  x  < h 


M  JS 


TS 


TS 


CDF  OF  THE 
COST  FUNCTION 


Fig.  4  -  Simpla  Cost  Function 


Y  *  a  +  bx17 
FOR  •  a  =  0 
b  •  1 
V  =  1 


Since  we  know  g  (y)  we  may  find 
•  oo 


and 


■>  •/> 

w  —  00 

(y)  -  / 

J  — < 


g(y)  dy. 


g  (y)  dy, 


so  A  and  M  moments,  as  well  as  a  distribution 
function  and  confidence  limits  may  be  found 
directly. 


We  have  found  this  form  almost  universally 
applicable.  Figure  5  provides  some  examples 
of  useful  transformations  that  often  appear  in 
cost  CERs.  Likewise  a  more  general  CER  of  the 


•  a  =  0  y  * 

b=  1 

•  a  =  0  Y=7 

b  *  -1 

•  a  -  0  y  *  VF 

b=>  1/2 

Fig.  5  •  Aimort  Universal  CER  Transformation 

The  general  form  of  the  computer  routine  we 
use  to  solve  most  problems  is  shown  in  Fig.  6. 
It  Is  a  simple  repetitive  procedure  using  only 
a  few  subroutines  and  the  addition  and  multi¬ 
plication  of  sets  of  moments. 
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_ _ _ 

COMPUTE  MOMENTS  FOR 
)  ALL  SIMPLE  AND 
1  COMPLEX  CERs 

J 


ADD  AND  MULTIPLY 
►  MOMENTS  ITERATING 
AS  REQUIRED 


1  COMPUTES  DISTRIBUTION 
l  CHARACTERISTICS  SKEW 
(  NESS  AND  KURTOSS 
J  FROM  A  MOMENTS 


COMPUTES  BETA  FUNC¬ 
TION  VALUES  AND  PLOTS 
PDF  AND  CDF 


Fig.  6  -  Generalized  Computer  Routine 


However,  it  is  possible  to  find  CERs  that  can 
not  be  handled  as  independent  elements  in  the 
power  form.  For  example: 

C  =  >/xT  +  2X1X2 


2  f 

C  =  aXx  +  d  X3 

Following  (18)  by  first  taking  a  first  or 
second  order  Taylor  Series  expansion  of  C  and 
calculating  the  first  four  A  moments  for  thia 
approximate  C. 


Expand  Y  ■  C  (X)  by  the  Taylor  series  around 
u: 


Y  -  C  (u)  +  (X-u)  C*  (u)  +  R 

disregarding  the  remainder  term,  and  taking 
expectations  of  both  sides  yields 

E  (Y)  *  C  (u)  +  0  »  dy 

Now  square  the  expansion: 

Y2  =  C2  (u)  +  2  (X-u)  C'  (u)  C  (u)  + 

(X-u)2  C'2  (u) 

then 

2  2  2  *2 
Y  -  C  (u)  -  (X-u) Z  C  (u)  + 

2  (X-u)  C  (u)  C  (u) 

again  taking  expectations  on  both  sides: 

E  (Y2)  -  E2  (Y)  »  O-^2  C  (u)  +  0 
by  definition: 

V  (Y)  =ax2  c'2  (u)  =  uy(2) 

This  may  be  expanded  to  a  vector  Y  * 

C  (X^,  X^  ...»  X^),  and  applied  to  find  the 

A  moments  (as  shown  in  (18)): 

A1  =  c  {UX’  U2’-"Un) 


=  !<• 


ac(x) 


dx. 

1 


x2  (2) 

u.  )  u. 

1  1 


u.)3  u.  (3) 
1  1 


<?:(x) . 

dX. 

1 


(2)  2, 


where  the  partials  are  evaluated  at  u^. 

The  second  order  expansions  are  more  complex 
(generally  adding  the  second  partials  terms) 
adding  additional  but  small  corrections  to  the 
first  order  approximations.  Considering  the 
noise  inherent  in  the  input  data  and  the  CERs, 
our  standard  practice  is  to  use  the  first 
order  approximation. 

This  procedure  can  not  be  made  a  standard 
subroutine  in  the  sense  that  the  pa  r  L la  I « 
must  be  computed  from  the  CERs  for  each  applica¬ 
tion.  As  these  CER  forms  are,  by  our  defini¬ 
tion,  fairly  rare,  partials  must  be  computed 
each  time. 
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OUTPUT  COST  DISTRIBUTIONS 


The  most  understandable  way  to  present  the 
output  appears  to  be  a  Cumulative  Distribu¬ 
tion  Function  (CDF),  even  though  the  Prob¬ 
ability  Density.  Function  (PDF)  is  more 
familiar  to  most  people.  The  CDF  is  the 
integral  of  the  PDF.  It  shows  for  any  arbi¬ 
trary  value  of  cost  the  probability  that  the 
cost  will  be  equal  to  or  less  than  the  arbi¬ 
trary  value  selected  (Fig.  7).  Generally, 
we  have  had  no  problem  fitting  the  final 
output  to  a  Beta  distribution,  though  we 
have  at  times  used  various  non-parametric 
Markoff  type  moment  approximations.  When 
comparisons  must  be  made  between  similar 
systems  or  between  high  gain  high  risk  ver¬ 
sus  low  gain  low  risk  systems,  we  have  no 
panacea.  We  feal  that  a  presentation  of  the 
CDFs  makes  the  issue  clearer  but  does  not 
make  the  decision  easier. 


AREA  =  A 


PR08A8IUTY  DENSITY  FUNCTION  (PDF) 


We  decided  to  use  the  Beta  distribution  as  our 
choice  of  an  output  CDF  for  two  reasons!  1)  it 
has  a  long  history  of  use  in  cost  uncertainty 
analysis  and  2)  it  is  fairly  "general"  in  the 
sense  that  it  can  satisfy  four  properties, 
whereas  the  exponential  satisfies  one;  Normal, 
ordinary  Gamma,  Weibul,  two;  and  so  on.  This 
concept  is  amplified  considerably  in  (19). 

Other  distribution  families  of  the  same  gener¬ 
ality  did  not  appear  to  be  more  useful  in  the 
parametrics  costing  field;  therefore,  we  decided 
to  represent  our  CDF  by  Beta  distributions. 

The  usual  form  of  th£  Beta  function  is 

f  (X)  -  C  z“  (1  -  zf 

0£Z<  1 


Slightly  different  variations  exist,  depending 
on  how  the  interval  is  defined.  For  probability 
functions,  C  is  selected  so  as  to  set  the  area 
under  the  curve  equal  to  1.  A  more  useful  form 
is  to  define  the  function  as  the  internal  A  to 
A  +  B  with  B>0.  Then 


,(x)  _  LLsJJLtll  ,  (2 

'  r  (  a+  1)  r  (3  +  1)B  \ 


where  A<  x<>  A+B,  ,  0  >  -  1  and 


tion 


ro  u  -  1 
z 


EXP  (-2)  dz 


(y)  *  Gamma  func- 


CUMULATIVE  DISTRIBUTION  FUNCTION  (CDF) 
Fig,  7  *  Relationship  of  PDF  to  CDF 


The  CDF  of  the  output  can  be  constructed 
from  the  four  moments  either  non-parametrically 
as  a  Tchebychef f-Markof f  four  moment,  fit  or  as 
a  Beta  function.  As  four  moments  provide  a 
closer  fit  for  the  function  than  three  or  two, 
we  compute  these  upper  and  lower  bounds  on  the 
CDF  following  (24).  Then,  we  rather  arbitrar¬ 
ily  assumed  that  the  CDF  may  be  approximated 
by  the  average  of  the  upper  and  lower  bounds. 
Figure  8  is  a  typical  CDF  constructed  from 
one  of  our  computer  graphics  routines,  com¬ 
paring  the  TM  approximation  of  the  Normal  and 
triangular  distributions  to  their  actuals. 


As  we  knew  that  the  Beta  distribution  was  part 
of  the  Pearson  system  of  curves,  we  turned  to 
Elderton  (4)  and  the  revised,  easier  to  read 
revision  by  Elderton  and  Johnson  (5)  to  see  if 
their  moment  fitting  approach  could  be  applied 
to  our  problem.  They  defined  Beta  parameters 
in  terms  of  the  moments  and  combined  the  moments 
into  the  coefficients  of  skewness  and  kurtosi«s. 
However,  they  used  some  clever  algebraic  substi¬ 
tutions  and  were  able  to  solve  the  non-linear 
equations.  The  algebraic  details  and  the  com¬ 
puter  programs  used  to  determine  a  and  &  are 
described  in  (I). 


APPLICATIONS 


We  use  risk  assessment  more  as  part  of  the 
preliminary  design  process  than  as  an  aid  in 
deciding  which  projects  to  procure  or  to  deter¬ 
mine  a  level  of  funding  for  a  series  of  invest¬ 
ments.  Therefore,  our  efforts  are  directed 
either  to  presenting  a  better  view  of  the  pro- 
ject_cost  picture  or  to  finding  the  cost  drivers 
that  cause  the  most  variation  in  this  picture 
so  that  further  action  can  be  taken. 

Although  we  feel  that  a  CDF  presents  a  good 
picture  of  possible  costs,  it  can  not  be  used 
in  isolation  to  justify  a  project  externally. 

It  is  either  too  wide,  meaning  "a  good  chance 
of  an  overrun",  or  too  narrow,  meaning  "add  a 
few  bucks  and__go  fixed  price"  _(Fig^9) . 
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TRIANGULAR  DISTRIBUTION 


NORMAL  DISTRIBUTION 


Fig.  8  -  TM  Approximation*  for  Popular  Distributions 


f 


RELATIVE  COST  {$)— ► 

(A)  HIGH  RISK  PROJECT 


RELATIVE  COST  ($)-* 
(B)  LOW  RISK  PROJECT 


Fig.  9  -  Typical  Program  Cost  Curves 
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Further,  when  several  projects  are  compared, 
it  is  still  not  easy  to  quantify  risk,  espe¬ 
cially  in  a  sense  of  helping  decision  makers 
accept  one  candidate  over  another  or  even  to 
accept  or  reject  a  single  project. 

Any  situation  of  clear  textbook  dominance  is 
so  apparent  that  no  manager  would  have  re¬ 
quested  a  risk  analysis. 

In  a  more  positive  sense,  we  use  the  CDF  to 
provide  a  rough  indication  of  how  bad  the 
situation  may  be  (Fig,  10),  isolating  those 
features  of  the  design  causing  the  most  vari¬ 
ance  in  the  CDF.  Once  isolated,  programs  are 
set  up  to  reduce  the  spread  by  further  analy¬ 
sis,  lab  tests,  experiments,  etc.  Thus,  a 
program  increases  some  in  cost  but  decreases 
in  risk  (Fig.  11).  This  is an  iterative  _ 


A  CLEAR  VIEW  OF  THE  COST  PICTURE 


Fig.  10  -  A  Typical  Output 


procedure  allowing  continuous  and  rational 
risk  reduction  during  the  initial  design 
process . 

A  more  likely  situation  is  shown  in  Fig.  12 
comparing  solar  powered  satellites  using  three 
solar  cell  material.  This  CDF  represents  the 
cumulation  of  over  a  100  inputs  processed 
through  about  50  fairly  complex  CERs.  As  can 
be  seen,  the  design  values  had  a  very  low 
probability  due  to  the  skewness  of  some  of  the 
major  causes  of  variation.  Using  this  ap¬ 
proach,  it  was  easy  to  determine  the  roost 
significant  contributors  to  the  variation  in 
output.  Needless  to  say,  they  were  not  read¬ 
ily  apparent  by  initial  inspection.  Also  of 
significance  was  the  relative  shapes  of  the 
cadmium  versus  silicon  nurves  which  made 
cadmium  appear  more  favorably  than  originally 
conj  ectured .  

A  further  application  applies  to  the  total 
uncertainty  contained  in  a  CER.  We  would  like 
to  show  how  the  method  of  moments  allows  us  to 
take  account  of  both  the  input  uncertainty  and 
the  statistical  uncertainty  of  a  regression 
developed  CER.  This  example  was  taken  from  a 
space  cost  model  [25]  and  is  deemed  represen¬ 
tative  of  this  type  of  situation. 

The  CER  for  first  unit  cost  of  the  structure 
of  a  thermal  control  and  interstage  subsystem 
is  gi^en  by: 

y  -  91.04  +  9.89  x  0,75 

where 

y  *  1976  dollars  in  thousands 

x  -  weight  in  pounds 
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(MATURE  SATELLITE" ' 
SMALL  CONSTRUCTION 
BASE,  CONSTRUCTION  AT 
LEO,  VARIOUS  SOLAR 
CELL  MATERIAL 


1.0  r 


Fig.  12  -  Candidate  SPS:  Cumulative  Distributions 


The  standard  error  of  the  estimate  is  given  as  * 

- -  *  t  : 

•  •  i  lo 

:  \ 


275.4. 

For  a  particular  application  the  point  design 
weight  was  given  as  234.6  and  the  selected 
values  for  the  triangular  distribution  were: 

Low  -  253.4 

Mode  -  351.9  * 

High  -  480.9 

Using  the  transformation  of  variables  the 
output  moments  are  shown  in  Fig.  13. 


This  standard  error  is  assumed  to  be  normally 
distributed,  with  a  mean  of  zero  and  standard 
derivation  of  275.4.  Thus,  we  have  two  inde¬ 
pendent  random  variables,  one  with  addition 
moments  as  given  above,  and  the  other  nq*ffi4lj__ 

with  moments  Al  -  0,  A2  -  275. 4^,  A3  -  A4  -  0. 
The  convoluted  distribution  then,  has  moments 
as  follows: 


First 


Al 


910.66 


Second  -  A2  -  82119.97 
Third  -  A3  -  31255.08 
Fourth  -  A4  -  -23953330 


I 


The  Beta  parameter  "of  this  distribution  sbow'a 
and  in  the  800's,  and  a  skewness  of  0..0O13, 
kurtosis  of  2.996,  hence  we  will  use  the  1 
normal  distribution  to  approximate  it. 


The  PDFs  and  CDFs  are  shown  in  Fig^  T5"I  As  is*- 

ioftlsn  the  case  when  the  CETRs  are  determined  by 
only  a  few  data  points  the  input  uncertainty 
seems  to  be  overwhelmed  by  the  statistical 
uncertainty.  This  result  should  be  recognized 
when  statistical  CERs  are  used,  and  due  caution* 
exercised  in  using  point  estimates  thus  based. 


X  TRIANGULA  l-l 
*>:r  0,rs 


MOMENTS  FOR  Ysfl+B*:<  +  N 
CEP  IS  Y*  91. 04  * 

351.9  *  480,9  > 

NOMINAL  VALUE*  394 .5659966 

ORIGIN  MOMENT Sv MULTIPLICATIVE > 
FIRST  *  910.658616 

SECOND*  335573. 9178 

THIRD  -  7723S2249.2 

FOURTH^  7.19167E+U 

ADDITIVE  MOMENTS 


X  TRIANGULAR 


FIRST 
SECOND"* 
THIRD  = 
FOURTH* 


V10. 658616 
6374, 8027S5 
31355.08 
-33953330 


FOURTH  CENTRAL  MOMENT = 


COEFFICIENT  OF  SKEWNESS* 
COEFFICIENT  OF  KURTOSIS* 


0. 063801 15? 
3.391633877 


Fig.  13  -  Computer  Output 
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SUMMARY 


We  feel  that  the  use  of  the  method  of  moments 
coupled  with  the  characterization  of  the 
inputs  by  triangular  distribution  is  a  viable 
alternative  to  the  use  of  Monte  Carlo.  Gener¬ 
ally,  it  is  more  accurate  and/or  requires  less 
computer  time.  However,  it  usually  requires 
more  analytic  effort  to  implement  and  is 
harder  to  explain  to  non-specialists  in  the 
f ield . 
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ABSTRACT 

This  paper  was  developed  given  the  assumption 
that  there  will  inevitably  be  delays  and  dis¬ 
ruptions  in  any  major  systems  acquisition. 
There  is  little  point  in  citing  here  a  long 
litany  of  problems  with  systems  acquisitions 
for  illustration.  It  (the  assumption)  is 
universally  recognized  as  an  acquisition 
"fact  of  life."  What  is  not  nearly  as  well 
known  is  what  to  do  about  the  effects  of 
delays  and  disruptions.  Specifically,  who  is 
responsible  and  who  pays  for  these  effects? 

The  government?  The  defense  contractor? 

This  paper  is  designed  to  review  a  methodology 
and  illustrate  its  application  as  one  way  of 
resolving  these  questions.  In  the  process, 
another  objective  is  to  advance  this  aspect 
of  program  management  closer  to  the  science 
of  management  as  the  basis  of  its  art. 

The  fundamental  theme  of  this  paper  is  that 
there  are  methodologies  availiabe  today  that 
can  help  program  managers  manage  delays  and 
disruptions  during  the  acquisition  process. 
This  paper  sets  the  stage  for  a  discussion  of 
one  of  these  methodologies--a  version  of 
simulation  called  system  dynamics — by  develop¬ 
ing  the  concept  of  delays  and  disruptions  in 
systems  acquisition  settings. 

This  commentary  is  followed  by  a  short  survey 
of  some  of  the  past  and  current  folkways  of 
trying  to  "manage"  delays  and  disruptions. 

The  authors  then  propose  system  dynamics  as 
one  methodology  for  monitoring  and  managing 
delays  and  disruptions  that  works. 

A  brief  discussion  of  system  dynamics  follows 
and  a  recent  successful  application  of  this 
methodology  in  systems  acquisition  is  descri  - 
bed.  The  paper  concludes  with  a  short  discus¬ 
sion  of  how  this  approach  to  delay  and  disrup¬ 
tion  management  might  be  implemented  and 
institutionalized  in  the  systems  acquisition 
process  for  a  large  project's  life  cycle. 

BACKGROUND 

A  review  of  past  issues  of  the  Aerospace  Daily 
included  a  number  of  items  of  interest  that 
set  the  stage  for  this  paper  U]  .  Among  those 
items  cited  were: 

"...the  60  Minuteman  II  missiles  covered  by 
the  Ford  Administration  supplemental  request 
will  not  be  built  and  the  FY  '77  funds  will  be 


"...The  B-l  program  was  cut  $280  million  in 
FY  '78— from  $2.15  billion  to  $1.87  billion— 
and,  as  expected,  the  number  of  bombers  to  be 
procured  fell  from  eight  to  five." 

"...F-15  production  was  cut  back  from  a  rate 
of  108  a  year  to  78.  The  $334  million  resul¬ 
ting  saving  would  be  applied  to  the  original 
$1.76  billion  request  to  bring  it  down  to 
$1.43  billion." 

"The  Sikorsky  CH-53  helicopter  for  the  Navy 
and  Marine  Corps  was  also  cut  by  $62  million 
(out  of  an  original  $87  million  request)  as 
part  of  a  move  to  stretch  out  the  program  by  a 
year . " 

"The  Army's  AAH  was  cut  in  half — from  $200 
million  to  $100  million — reflecting  what  DOD 
called  a  more  thorough  evaluation  of  options 
and  what  (Secretary  of  Defense)  Brown  more 
specifically  identified  as  a  concern  over  the 
helicopter's  vulnerability." 

"In  the  shipbuilding  area,  the  Aegis-equipped 
CSGN  was  knocked  out  entirely  for  a  $187  mil¬ 
lion  reduction  while  another  $43  million  was 
deleted  for  conversion  of  an  LST  to  a  PHM. 

The  PHM  program  will  be  further  reviewed." 

There  are,  of  course,  many  more  examples  that 
could  be  quoted,  in  areas  including  funding 
increases,  performance  upgrading,  schedule 
stretchouts,  and  the  like.  Over  and  over 
again,  events  seem  to  overtake  program  mana¬ 
gers  and  produce  dramatic  impacts  on  system 
costs,  performance,  and  schedules.  Over  and 
over  again,  delays  and  disruption^,  both 
planned  and  unplanned,  intended  and  uninten¬ 
ded,  seem  to  haunt  program  managers.  Yet,  as 
has  been  said  before,  there  is  nothing  new 
about  this  state  of  affairs. 

DELAYS  AND  DISRUPTIONS 

Delays  and  disruptions  hereafter  referred  to 
as  D&D's,  is  merely  a  new  concept  for  a  tra¬ 
ditional  issue.  The  problems  caused  by  D&D's 
have  existed  since  man  was  first  confronted 
with  the  management  of  others  and  his  resour- 
ses  in  relation  to  the  environment.  The  Ro¬ 
man  Aqueduct  had  a  cost  overrun  of  over  100%. 
The  Suez  Canal  a  cost  overrun  of  200%.  The 
Panama  Canal,  built  by  the  French  because  of 
their  experience  at  Suez,  nevertheless  had  an 
overrun  of  over  70%  on  cost  [9]. 
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In  fact,  during  the  first  Systems  Engineering 
and  Management  Seminar  recently  held  in  the 
People’s  Reoublic  of  China  (PRC)  (in  the  spring 
of  1980  in  Peking),  one  of  the  authors  (Sherman) 
in  addressing  the  officials,  faculty,  and 
researchers  of  the  aeronautics  and  astronautics 
communities  complimented  the  audience  on  the 
fantastic  management  and  engineering  feat  of 
the  Chinese  in  building  the  Great  Via  1 1  [29]. 
This  wall,  according  to  our  astronauts,  was 
the  only  man-made  object  on  earth  clearly 
visible  from  outer  space.  The  Great  Wall  was 
started  in  the  4th-5th  century  and  the  effort 
ended  in  the  15th— 16 th  century.  It  was  over 
3,100  miles  in  length  (at  one  time)  and  at  its 
peak  construction  period  emoloyed  over  300,000 
persons  for  a  ten-year  period. 

However,  the  audiences’  pride  was  soon  tempered 
when  it  was  noted  that  there  were  three  basic 
problems  with  this  feat  from  the  oersoective 
of  American  orogram  managers:  (1)  the  Great 
Wall  was  not  completed  on  time,  (2)  the  cost 
overrun  was  fantastic  and  acceotable  only  to 
enoerors ,  and  (3)  the  wall  never  Performed  the 
function  for  which  it  was  built.  In  terms  of 
cost,  performance,  and  schedule,  the  Great 
Wall  was  a  great  disaster. 

It  seems  that  delays  and  disruptions  are  des¬ 
tined  to  impact  system  costs,  schedules,  and 
performance.  If  this  is  so,  we  might  try  to 
place  D&D's  in  context.  Figure  1  shows  the 
linkage  between  changes,  D&D's  and  impacts  on 
cost,  schedules,  and  performance, 

!cost  imoacts 
schedule  impacts 
performance  impacts 

Figure  1.  Linkage  between  Changes,  D&D's  and 
System  cost,  schedule  and  perfor- 
ance  impacts. 

Changes  produce  D&D's.  Some  changes  are  beyond 
the  control  of  either  the  government  or  the 
defense  contractor  while  others  may  be  ini¬ 
tiated  by  either  or  both.  We  present  here  a 
few  examnles  to  illustrate  the  point. 

SOME  CAUSES  OF  DELAYS  AND  DISRUPTIONS 

The  causes  of  delays  and  disruptions  were 
classified  in  the  pioneering  work  of  E.3. 
Cochran  E  8  i  This  section  relies  heavily  on 
his  work  and  research  into  disruotions  in 
major  acquisition  programs.  Figure  2  presents 
this  taxonomy  of  sources  of  disruption. 


Internal  Planning  and  Management 

One  cause  of  D&D's  is  chances  due  to  internal 
planning  and  management;  for  example,  planning 
or  errors  in  cost  estimating.  First,  cost 
estimating  is  at  best  an  art,  not  a  science, 
especially  in  the  early  stages  of  the  systems 


acquisition  process.  It  should  come  as  no 
surorise  that  unexoectedly  high  inflation, 
suonly  and  demand  factors,  ooor  resource  allo¬ 
cation  and  managerial  inefficiency  can  cause 
total  estimated  program  costs  at  completion 
to  exceed  early  estimates  by  10,  100,  or  even 
300  percent  [34]. 

A  more  recent  addition  to  the  list  of  inaccu¬ 
rate  planning  estimates  involves  the  entire 
man-machine  system  of  a  weapons  system.  There 
has  been  a  substantial  droo  in  the  capabili¬ 
ties  of  the  manpower  available  to  the  services 
in  recent  years  (21*3.  Consequently,  system 
changes  are  being  forced  because  either  the 
operators  or  the  maintenance  force  are  incao- 
able  of  performing  at  the  levels  for  which 
the  system  was  initially  designed.  For 
example,  some  eauioment  in  the  surface  fleet 
is  not  used  at  all  because  the  talents  re¬ 
quired  to  operate  and  maintain  it  exceed 
those  available  in  current  sailors  [21].  And 
the  problem  is  not  unique  to  the  U.S.  Navy. 

Finally,  another  example  of  the  management 
cause  is  that  the  defense  contractor  may 
simply  have  "bought  in"  with  the  initial  bid. 
In  this  case  the  contractor  is  relying  on  the 
usual  assortment  of  changes  that  are  bound 
to  occur  to  provide  the  opoortunity  to  make 
up  the  effects  of  the  "buy  in"  or  subsequent 
mismanagement. 

External  Forces 

Secondly,  there  are  external  forces.  For 
example,  in  order  to  keep  the  total  program 
costs  within  some  acceptable  range  for  con¬ 
gressional  review,  initial  costs  (and  often 
specifications,  quantity,  and  so  on)  may  be 
underestimated  with  the  tacit  understanding 
that  "system  growth"  can  be  managed  during 
oroduction  with  change  orooosals  to  bring  the 
system  up  to  the  desired  capabilities  (pluss¬ 
ing  up!).  In  effect,  this  is  an  intentional 
end  run  of  the  limitations  and  constraints 
of  the  DSARC  process  called  for  in  DoD  Direc¬ 
tives  5000.1  and  5000.2.  The  assumption  is 
that  it  is  easier  to  get  what  you  want 
through  change  proposals  once  in  production 
than  to  get  what  you  want  in  the  Milestone  I, 
II,  and  III  decisions.  And  this  assumption 
is  probably  correct. 

The  impacts  of  newly  generated  federal,  state 
and  local  regulations  and  standards  (after 
award)  can  be  highly  cost  escalating  and 
cause  schedule  stretch  out.  Disposal  of 
toxic  wastes,  new  occupational  safety  and 
health  rules,  and  other  environmental  actions 
for  air  and  noise  pollution  are  familiar  ex¬ 
amples  that  frequently  require  mandated  capi¬ 
tal  investments  [  13  3.  Other  examples  are 
major  program  changes,  price  escalations  and 
force  majeures . 
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Figure  2  Sources  of  Project  Disruption 
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Concurrency 

Third  is  the  concurrency  cause.  If  a  system 
is  the  first-of-a-kind  as  most  new  hardware  or 
software  is,  the  specific  system  requirements 
may  not  be  well  defined  (technological  uncer¬ 
tainty)  at  the  time  of  contracting.  Hence, 
this  will  inevitably  lead  to  many  changes 
(some  significant)  in  the  system  while  in 
development  and  oroduction.  Delivery  urgency 
and  interrelatedness  are  also  exampl  es  of  the 
concurrency  cause. 

Concurrency  Defined 

The  author's  use  of  the  word  concurrency  is 
different  and  broader  from  the  commonly  held 
DoD  and  USAF  philosophy.  The  DoD  uses  the 
term  to  describe  "planned"  concurrency  where 
the  stages  of  the  acquisition  life  cycle  in¬ 
tentionally  overlao  to  accelerate  delivery. 

An  example  is  when  development  overlaps  with 
the  initiation  of  oroduction  activities  to 
compress  time,  as  in  some  early  missile  pro¬ 
grams  . 

Our  use  of  concurrency  includes  the  case  where 
uncertainty  forces  unplanned  overlap  in  the 
stages  of  the  acquisition  life  cycle.  In  this 
case  concurrency  also  occurs.  Recognizing  and 
accepting  this  use  of  terminology  is  useful  in 
understanding  DSD's. 

Factors  of  Concurrency 

The  basic  factors  of  concurrency  are  shown  in 
Figure  3.  Delivery  urgency 

(U)  is  usually  market  generated  by  a  particular 
customer,  or  competitive  conditions  where  a 
firm  commits  to  an  inherently  optimistic 
delivery  date.  As  delivery  urgency  develoos, 
the  initial  delivery  is  usually  eased  thereby 
generating  unplanned  cost  increases. 

In  addition,  the  firm  usually  assumes  that  no 
serious  problems  in  technology  will  arise  if 
the  effort  is  state-of-the-art  (SOA)  or  even  a 
modest  advance  (SOAA).  Our  definition  of  SOA 
(A  in  Fig.  3)  also  differs  from  the  general 
use  in  DoD.  For  example,  combining  existing 
black  boxes  (off-the-shelf)  is  not  SOA  when 
used  in  any  new  or  different  combination, 
package,  hardware  or  system.  SOA  is,  accord¬ 
ing  to  the  authors,  procuring  an  existing 
production  line/warehouse  product  for  use  as 
is,  or  as  designed,  with  no  other  useor  appli¬ 
cation.  If  not,  then  what  really  occurs  is 
some  art  advance  (SOAA). 

The  normal  statements  on  SOA  generally  do  not 
reflect  the  varying  degrees  of  technical  un¬ 
certainty  existing.  This  technological  uncer¬ 
tainty  (T)  is  intensified  by  the  degree  of 
interrelatedness  (I)  in  the  design  and  pro¬ 
duction  process.  Where  there  is  a  higher 
interrelatedness,  there  is  increased  uncertain¬ 
ty. 


If  the  SOAA  is  a  fundamental  driver  of  tech¬ 
nological  uncertainty,  then  the  interrelated¬ 
ness  is  the  major  multiplier  of  the  imoacts 
on  the  costs  of  develooment  and  production. 

Other  factors  compounding  the  problem  and 
generating  additional  concurrency  are  the 
level  of  resources  (R)  and  the  degree  of  ex¬ 
ternal  control  (X)  over  events. 

D&D  Process 

Concurrency  develops  and  causes  DSD's  in  many 
ways,  often  unrelated  in  advance.  In  Figure 
4,  a  static  model,  or  flow  diagram  of  con¬ 
currency  disruption  for  a  broad  based  program 
focusing  on  the  orocess  of  DSD’s  and  identi¬ 
fying  cost  impact  ooints  is  illustrated 
(shown  as  dotted  line  boxes). 


A  firm  normally  starts  a  program  that  requires 
the  delivery  of  a  specific  product  within  a 
specified  period  of  time.  While  judged  SOA 
it  is  not  fully  designed,  although  the  pro¬ 
duct  is  familiar  (as  is  the  process)  to  the 
firm.  Serious  technical  problems  develop. 

They  require  extensi ve  time  to  resolve.  Sub¬ 
stantial  changes  to  the  original  design  or 
tasks  are  made.  In  turn  this  generates  a 
near  complete  replanning  effort  including 
schedules  and  production  procedures.  Costs 
scar  as  D&D’s  occur. 

Now  intensive  efforts  to  minimize  the  devia¬ 
tion  from  the  original  delivery  dates  inter¬ 
act  with  the  required  changes  in  schedules 
and  production  procedures.  Costs  increase, 
and  confusion  is  caused  by  the  changes  that 
are  now  compounded  by  the  compression  of  time. 
Time  is  compressed  by  the  delays  and  added 
tasks  as  new  constraints  are  introduced.  The 
tight  delivery  schedules  create  a  great  deal 
of  the  problem. 

All  this  intensifies  the  effects  of  change  on 
cost.  Are  these  costs  normally  tracked  and 
accounted  for  in  relation  to  causal ty? 

Effects  of  Concurrency 

Figure  5  illustrates  the  effects  of  concurren¬ 
cy  on  program  costs.  The  upper  life  cycle 
indicates  a  fairly  orderly  incremental  ap¬ 
proach  or  olan  that  is  generally  sequential 
with  little  or  no  overlap.  The  crosshatching 
is  added  costs  from  false  starts,  rework  and 
so  on. 


The  life  cycle  model  in  the  lower  half  of  the 
figure  shows  the  effects  of  concurrency 
caused  by  overlaps  generated  by  continuing 
uncertainty.  The  bulk  of  the  added  costs  are 
always  in  the  oroduction  stage  (or  phase). 
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Figure  3  Factors  in  Concurrency 
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CONCURRENCY  DISRUPTION 


Figure  4  Concurrency  Disruption 
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Figure  5  Effects  of  Concurrency 


D&D  Model 

Finally,  to  depict  the  D&D  concept  the  Cochran 
model  of  disruption  is  presented  as  Figure  6. 
This  was  designed  as  a  speculative  model  to 
present  the  impact  visually  of  the  disruption 
factors  (or  variables)  on  time  and  cost.  Dis¬ 
ruption  is  shown  on  a  curved  surface  with  the 
values  increasing  along  lines  U  and  T.  One 
cost  surface  is  defined  for  each  given  ratio 
of  resources  to  tasks.  As  the  resource  ratio 
raises,  the  cost  surface  rotates  downward. 
While  the  model  was  not  developed  from  general 
functional  characteristics,  there  is  an  oppor¬ 
tunity  for  research  to  continue  this  effort. 


D&D  Summary 

There  are  a  number  of  additional  categories  of 
changes  within  the  three  major  causes  that 
produce  D&Ds.  In  the  Rowe  and  Somers  paper  a 
fourth  major  cause,  customer  uncertainty,  has 
been  treated  separately  [  27  ].  That  is  not  the 
point.  The  point  is  that  program  managers  and 
defense  contractors  must  face  at  one  time  or 
another  D&D's  in  the  form  of  program  change 
proposals  (PCPs),  engineering  change  proposals 
(ECPs)  and  contract  supplemental  agreements. 

One  naval  shipyard  experienced  35,000  changes 
in  the  process  of  constructing  one  aircraft 
carrier.  And  each  change  can  produce  D&Ds 
that  have  the  potential  to  influence  cost, 
performance  and  schedule.  And  if  D&D’s  in  any 
of  their  various  forms  do  impact  costs,  per¬ 
formance  and  schedule,  who  absorbs  the  cost  or 
who  pays  the  bill? 

Who  pays  the  bill  is  not  too  much  of  a  problem 
if:  (1)  the  D&Ds  can  be  traced  back  to  changes 
either  under  the  control  of  the  government  or 
under  the  control  of  the  defense  contractor, 
and  (2)  the  exact  magnitude  of  the  cost,  per¬ 
formance  and/or  schedule  impact  can  be  allo¬ 
cated  to  each  D&D.  Since,  traditionally,  this 
tracing  and  identification  has  been  difficult 
to  accomplish  (or  not  attempted  at  all),  many 
disputes  have  occurred  and  program  managers 
(and  their  sponsors)  have  attempted  a  number 
of  methods  and  ways  to  squirm  out  of  resolving 
this  continuing  program  management  issue. 

SOME  FOLKWAYS  OF  COPING  WITH  DELAYS 

AND  DISRUPTIONS 

Ignorance  is  B1  i ss 

One  way  to  deal  with  D&D's  is  to  pretend  they 
don't  exist.  Then  if  they  do  occur  and  their 
impacts  can  no  longer  be  ignored,  the  govern¬ 
ment  simply  pays  the  bill,  revises  schedules, 
and/or  adjusts  system  performance  with  or  with¬ 
out  a  fuss  where  the  degree  of  fuss  is  propor¬ 
tional  to  media  exposure  and  congressional  com¬ 
plaints. 


The  Deep  Freeze 

In  this  approach  there  is  some  recognition  that 
the  root  of  D&D  problems  is  in  changes.  The 
solution  is  to  attempt  to  freeze  designs  and 
eliminate  ECP ’s,  PCP’ s and  the  like  after  some 
predetermined  point.  The  failure  of  this  ap¬ 
proach  is  borne  out  by  discussions  of  the  14th 
change  to  the  seventh  version  of  the  third  mod- 
fication  to  the  sixth  final  absolute  frozen 
plan. 

Competitive  Bidding 

Our  contracting  agencies  have  tried,  and  con¬ 
tinue  to  try,  competitive  bidding.  The  histor¬ 
ical  results  for  whatever  reasons  on  major  sys¬ 
tems  in  particular  have  been  viewed  in  the  pub¬ 
lic  and  the  congress  as  ’’buying  in.  V  This, 
again,  is  probably  due  to  an  underestimate  of 
the  D&D's (or  a  failure  to  accept  the  occurrance 
of  D&D’s)  likely  to  be  encountered  and,  once  en¬ 
countered,  the  attempts  to  up  the  price  tag  ac¬ 
cordingly. 

Although  the  history  of  system  acquisition  and 
program  management  since  the  Korean  War  con¬ 
tains  many  examples  of  government  sponsored  in¬ 
novations  in  contracting  techniques,  these  have 
been  met  by  contractor  responses  designed  to 
either  maximize  gain  or  minimize  losses 0  The 
greatest  opportunity  for  the  entire  community 
is  thought  to  lie  in  recognizing  and  admitting 
that  we  operate  under  the  free  enterprise 
system  and  that  competitive  bidding  should  be 
the  best  insurance  policy  against  the  effects 
of  D&Ds. 

Yet,  in  an  economic  sense,  the  market  place  is 
monopsonistic .  Essentially,  one  buyer  (feder¬ 
al)  and  a  few  qualified  sellers  exist.  Given 
these  conditions,  once  an  award  is  made,  a  con¬ 
tractor  may  act  as  a  monopolist  (or  quasi-mono¬ 
polist)  CZ9J  . 

One  example  illustrates  what  may  be  a  classic 
case  of  good  management  from  a  defense  contrac¬ 
tor  (and  government  program  manager)  point  of 
view.  That  is  the  case  of  the  McDonnell- 
Douglas  F-4  Phantom.  The  continual  modifica¬ 
tions  and  changes,  mission  expansions,  and  ap¬ 
plications  ranging  from  the  Navy  to  Air  Force 
resulted  in  many  design  changes  with  approved 
ECPs.  This  permitted  McDonnell-Douglas  to 
maintain  a  ’’non-loss’’  production  line  for  many 
years  while  still  maintaining  state  of  the  art 
technology.  In  fact,  the  product  differentia¬ 
tion  concept  of  the  consumer  goods  marketplace 
was  successfully  applied  on  the  F-4  for  market 
maintenance. 

There  are  many  examples  where  the  results  of 
competitive  bidding  have,  on  the  other  hand, 
produced  less  than  satisfactory  results.  In 
fact,  some  acquisition  specialists  have  sug¬ 
gested  that  those  qualified  defense  contrac- 
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tors  simply  be  listed  alphabetically  and  each 
be  awarded  in  turn  the  next  defense  contract 
up  in  their  area  of  expertise.  Others  have 
noted  the  development  of  symbiotic  relation¬ 
ships  to  perpetuate  traditional  contractors 
[21]  . 

Government  Managed  Program  s 

The  NASA  manned  space  program  that  led  to 
landing  men  on  the  moon  was  a  competitively 
bid  program.  A  doctoral  dissertation  was 
written  on  the  factors  leading  to  changes  in 
cost  estimates  by  NASA  (and  active  DoD  pro¬ 
grams)  [l 8] ,  However,  contractors  were  in  ef¬ 
fect  subcontractors  to  NASA  who  maintained  sys¬ 
tem  program  management  responsibility.  Strate¬ 
gies  varied  from  extremely  close  ECP  cost- 
control  through  a  computerized  approval  and 
billing  system  at  North  American  (Downey)  to 
the  Grumman  approach.  Grumman  was  responsible 
for  the  LEM  (Lunar  Excursion  Module)  and  appar¬ 
ently  did  not  establish  such  a  complete  system 
(partly  caused  by  their  long  experience  and 
monopoly  relationship  with  a  primary  customer — 
the  Navy).  While  Grumman  developed  a  success¬ 
ful  product  from  a  performance  standpoint,  the 
cost  of  the  system  may  not  have  provided  for  a 
complete  or  adequate  financial  recovery  by  the 
firm. 

Contracting  new  systems  and  programs  other  than 
by  the  federal  government  is  not  without  over¬ 
run  problems.  For  example,  the  BART  system  in 
San  Francisco  had  a  three  year  schedule  delay, 
a  $584  million  cost  overrun  (60%),  and  a  per¬ 
formance  efficiency  noted  at  one  time  at  less 
than  50%  [4]. 

Multiple  Incentive  Contracting  Approach 

In  the  1960’s  multiple  incentive  contracting 
was  the  DoD  program  response  to  D&D's  as  the 
TFX  (F-lll)  was  contracted  for  a  multi-service 
buy  and  operation  [35].  The  contractor  stra¬ 
tegy  after  award  was  to  optimize  incentive  dol¬ 
lars  by  concentrating  on  Time  and  Cost  incen¬ 
tives  to  meet  scheduled  deliveries  and  budgeted 
program  costs,  and  by  failing  to  deliver  on 
performance.  The  strategy  was  not  difficult 
for  decision-makers  when  it  became  apparent 
that  the  engine  subcontractor  could  not  meet 
performance  criteria,  and  the  Navy  had  required 
changes  in  structure  and  thus  weight.  These 
D&D's  would  not  permit  attainment  of  the  incen¬ 
tive  fee  for  meeting  the  contract  performance 
criteria.  Thus  management  emphasis  shifted  to 
time  and  cost  controls. 

Fly  Before  Buy  Approach 

The  concept  of  "fly  before  buy”  was  another 
Pentagon  solution  (although  tried  until  1957) 
[21]  to  the  D&D- induced  cost  overrun  problem 
[36],  It  was  thought  that  a  ”f lying"  version 
before  a  production  decision  could  reduce  the 


need  for  post  production  changes  and,  hence, 
reduce  D&D’s.  The  contractor  strategy  was  to 
gain  support  wherever  possible  (including  the 
purchaser)  and  try  to  influence  the  criteria 
for  the  decision  making  model  in  the  test  and 
evaluation  phase  (fly  off).  The  result  appears 
to  have  been  successful  in  the  case  of  YA-10 
versus  A-7  fly-off  but,  did  the  Air  Force  real¬ 
ly  buy  the  best  tactical  fighter  given  the  real 
world  missions? 

This  may  not  have  been  as  apparent  with  ALCM 
but,  it  was  discouraging  to  many  that  the  "fa¬ 
vored"  contractor  experienced  more  initial  fai¬ 
lures  in  the  ten  (10)  vehicle  tests  [6  j.  And 
if  this  system  performance  was  lacking,  could 
this  be  corrected  by  ECP' s ,  PCF'  a  ,  e tc?  Would 
not  this  lead  right  back  to  D&D's? 

Consortia  Approach 

Consortia  of  firms  have  been  formed  as  a  stra¬ 
tegy  for  managing  the  D&D's  of  new  large  scale 
systems.  Complete  new  organizations  have  been 
created  but,  they,  also,  have  been  ineffective 
in  meeting  schedules,  costs,  and  performance 
goals.  For  example,  with  the  Alaska  pipeline, 
the  oil  companies  involved  (and  the  Fluor  Corp) 
formed  the  Aleyeska  Corp.  They  experienced  a 
time  or  schedule  overrun  of  five  (5)  years, 
a  cost  overrun  of  6.8  billion  dollars  from  the 
original  cost  estimate  (900  million  dollars). 
While  the  original  overrun  was  about  800%  (on 
cost),  the  figure  usually  quoted  is  the  revised 
cost  estimated  after  the  environmental  settle¬ 
ments  or  a  cost  overrun  of  "only  126%  (adjusted 
also  for  inflation)  [26 ]  . 

The  consortia  of  HBH  (Hughes-Bendix-Holmes/Nar- 
ver)  for  the  SNAP  program,  while  initiated  only 
in  1979,  is  already  experiencing  D&D’s.  The 
human  resources  are  not  available  to  recruit  and/ 
or  train  in-country  for  the  crews  to  man  the 
navy  under  construction  for  a  Middle  East  coun¬ 
try  [19]. 

Private  Development 

'In  the  private  sector  the  marketplace  is  not 
without  its  examples  of  corporate  overruns  cau¬ 
sed  by  D&D.  The  Lockheed  L-1011  with  the  pro¬ 
blems  of  Rolls  Royce  on  the  RB-211  engine  resul¬ 
ted  in  the  bankruptcy  of  RR  and  the  near  bank¬ 
ruptcy  of  Lockheed  Corporation.  The  Lockheed 
strategy  was  to  engineer  a  federally  insured 
bail-out  [28],  We  have  yet  to  see  whether  this 
is  successful,  although  the  technique  has 
since  been  applied  to  the  city  of  New  York  and 
the  Chrysler  Corporation. 

Total  Program  Package  Approach 

The  DoD  moved  to  a.  new  contract  in  g  concept  or  ap¬ 
proach,  TPP  (Total  Procurement  Package).  The 
Air  Force  applied  TPP  to  the  SCRAM  (ATG-69A) 
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program,  and  this  resulted  in  a  cost  overrun  of 
$297  million  or  200%  [32].  This  was  another  case 
where  TPP  was  applied,  yet  the  program  was  not 
a  major  improvement  in  philosophy,  but  in  reali¬ 
ty  an  attempt  to  advance  the  state  of  the  art 
significantly  in  solid  state  rocket  motors.  In 
addition  to  other  technological  breakthroughs 
necessary,  the  missile  was  also  to  be  deployed 
with  the  FB-111  (concurrently  under  development) 
and  with  later  versions  of  the  B-52,  and  become 
a  critical  component  of  the  Air  Force  arsenal. 

Lockheed  experienced  substantial  dollar  losses 
on  the  engine  development  for  this  program  and 
the  C5A  [7  ]. 

The  TPP  concept  using  a  FFF  (Firm  Fixed  Price) 
contract  was  also  applied  on  the  Ingalls  Ship¬ 
building  division  of  Litton  Industries,  Inc. 
contract  for  the  LHA  (amphibious  assault  ships) 
in  1969  and  the  DD— 963  (Spruance-class  destroy¬ 
ers)  [12]. 

The  LHA  program  experienced  a  complex  combina¬ 
tion  of  D&D?  s ,  resulting  in  a  $2.7  billion  cost 
overrun.  The  contractors  strategy  was  to  try 
to  maintain  schedule  and  performance  while  con¬ 
tinuing  cost  negotiations.  This  strategy  resul¬ 
ted  finally  in  legal  action  with  claims  of  seve¬ 
ral  billion  dollars. 

This  lawsuit  involved  cost  determinations  that 
were  complex  and  defied  simplistic  solution.  It 
was  ultimately  resolved  with  the  development 
of  a  production  model.  The  D&D's  were  traced 
back  and  the  identifying  causes  and  related 
costs  generated. 


trade-off  studies  may  at  that  moment  be  prima¬ 
rily  seat-of-the-pants  oriented  and  rely  on  the 
savvy  of  the  proponent  of  the  change  and  the 
program  manager,  they  are  still  probably  done. 
If  the  perceived  benefits  outweigh  any  estima¬ 
ted  adverse  impacts,  the  ECP  goes  forward.  And 
it  should.  The  only  question  is:  Who  pays  for 
it? 

The  only  problem  with  answering  this  question 
is  that  true  impacts  of  change-induced  D&D's 
are  hard  to  trace  through  the  system  acquisi¬ 
tion  process;  let  alone  to  link  them  explicitly 
and  numerically  to  cost,  schedule  and  perfor¬ 
mance  impacts.  And  there  is  the  rub.  Who  is 
going  to  determine  which  impacts  or  portion  of 
impacts  are  linked  to  which  changes  (D&D's) 
and  how?  Once  that  is  done  and  agreed  upon, 
we  assume  it  becomes  fairly  straight  forward 
to  figure  out  who  is  going  to  pay  for  them. 

We  are  proposing  a  new  way  to  manage  D&D's  that 
is  designed  to  answer  those  questions  and  oth¬ 
ers  like  them.  What  we  are  proposing  is  an 
emphasis  on  managing  D&D’s  through  simulation 
technology  currently  available. 

Through  this  methodological  approach  changes 
that  cannot  be  controlled  can  still  be  managed. 
Controllable  changes  can  be  explicitly  examined 
before  their  implementation  to  examine  their 
D&D's  with  their  resulting  impacts  on  program 
costs,  schedules  and  performance.  And  for  any 
change-induced  D&D,  the  impacts  can  be  deter¬ 
mined  explicitly  and  numerically  so  that  who 
pays  for  what  can  be  determined. 


Using  this  production  model  the  dispute~was 
finally  settled  with  a  financial  settlement  to 
'the  contractor  of  $447  million.  It  is  this 
last  approach  that  we  will  examine  in  the  re¬ 
maining  portions  of  this  paper  since  it  repre¬ 
sents  — to  us  —  a  significant  potential  to 
shift  emphasis  in  dealing  with  D&D's  in  pro¬ 
gram  management. 


MANAGING  DELAYS  AND  DISRUPTIONS 


In  the  context  of  the  foregoing  discussions, 
the  central  theme  of  this  paper  can  now  be 
stated.  We  are  proposing  that  program  managers 
shift  from  either  trying  to  ignore  or  prevent 
(by  freezing  designs  or  by  contracting  devices) 
D&D's  to  simply  accepting  D&D's  and  learn  how 
to  manage  them. 

In  fact,  rather  than  suggest  that  D&D's  are  all 
bad,  it  may  well  be  that  some  of  them  are  high¬ 
ly  desirable. 

For  each  D&D  produced  by  an  ECP,  for  example, 
someone  must  have  performed  some  kind  of  a 
trade-off  analysis  on  the  benefits  of  the  ECP 
compared  to  the  impacts,  if  any,  on  costs,  sch¬ 
edules,  and  system  performance.  While  these 


The  approach  we  are  advocating  is  a  specific 
version  of  simulation  called  system  dynamics 


SIMULATION  AND  SYSTEM  DYNAMI CS 


wul  d^jjtoacn  in  cnis  paper  is  tnat  ot  simula¬ 
tion  and,  more  specifically,  a  version  of  simu¬ 
lation  called  system  dynamics.  Simulation  has 
been  around  a  long  time,  and  modelers  who  use 
simulation  have  generally  followed  a  two-step 
process.  First,  they  construct  a  model  of  some 
system  of  interest  to  them  in  order  to  imitate 
the  system's  behavior.  If  the  model  can  suc¬ 
cessfully  imitate  the  system’s  behavior  under 
a  wide  range  of  circumstances,  the  modeler  will 
feel  confident  enough  to  move  on  to  the  second 
step:  running  policy  experiments.  In  this 
step  the  modeler  alters  particular  inputs  to 
the  model  that  correspond  to  real  world  policy 
options.  For  example,  the  Congress  could  cut 
the  budget  of  a  program  by  20%.  By  interpre¬ 
ting  the  model's  responses  to  the  changes,  the 
modeler  hopes  to  understand  how  the  real  system 
would  respond  to  the  same  policy  changes.  Once 
a  model  has  been  developed  and  confidence  has 
been  gained  in  its  utility,  the  modeler  can  ma¬ 
ke  his  model  available  to  interested  users  (for 
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example,  a  program  manager,  if  the  model  is  one 
of  a  system  for  which  the  manager  has  responsi¬ 
bility)  .  The  user  then  can  test  out  real  sys¬ 
tem  reactions  to  changes  using  his  model  of  the 
real  system  as  a  surrogate  for  it  [10,  3l]  . 

In  physical  systems,  simulations  are  relatively 
common  and  run  the  gamut  from  simple  hand  simu¬ 
lations  with  tabletop  models  and  physical  mock- 
ups  to  more  recent  and  exotic  mathematical  mo¬ 
dels  and  computer  simulations  such  as  those 
used  to  control  space  flights. 

In  biological  systems,  experts  are  now  routine¬ 
ly  at  work  constructing  computer  based  simula¬ 
tion  models  of  kidneys,  cardiovascular  systems, 
etc.,  to  test  new  medical  policies.  In  social 
systems  (and  the  weapon  procurement  system  is 
a  social  system) ,  especially  over  the  past  de¬ 
cade,  there  has  been  a  great  increase  in  the 
resources  allocated  to  building  interdiscipli¬ 
nary  computer  simulation  models  of  complex  so¬ 
cial  systems.  In  each  area  of  application  the 
process  is  the  same.  A  model  is  constructed 
and  then  ’’tweaked"  to  see  what  kinds  of  respon¬ 
ses  are  produced. 

With  the  advent  of  large  digital  computers, 
modelers  have  turned  increasingly  to  developing 
models  of  complex  systems  that  are  programmable 
on  digital  computers.  In  fact,  most  current 
simulation  efforts  depend  heavily  upon  the  use 
of  a  computer  to  assist  in  manipulating  the 
otherwise  unwieldy  mass  of  data  and  interrela¬ 
tionships  that  have  to  be  considered. 

We  should  stress  that  using  a  computer  does  not, 
per  se,  lend  any  particular  authority  or 
correctness  to  an  analysis.  The  computer  is 
merely  a  very  fast,  very  accurate,  electrical 
idiot  that  manipulates  data  according  to  the 
framework  and  ground  rules  established  by  the 
model.  Should  the  computer  output  fail  to  ac¬ 
curately  initiate  the  behavior  of  the  real 
system,  the  model  and  the  modeler  (not  the  com¬ 
puter)  are  to  blame. 

While  the  advent  of  bigger  and  faster  computers 
has  helped  the  modeler  to  manipulate  larger 
bodies  of  information,  better  computers  are  not’ 
the  main  reason  for  the  growing  use  of  simula¬ 
tion  models.  Rather,  the  growth  in  the  com¬ 
plexity  of  the  systems  we  are  called  upon  to 
manage  and  the  increasing  need  for  an  inexpen¬ 
sive  means  to  test  out  policies  prior  to  imple¬ 
mentation  have  been  the  main  stimulants.  No 
engineer  would  dream  of  trying  to  analyze  and 
predict  the  behavior  of  complex  physical  sys¬ 
tems  (such  as  those  used  in  space  flight  con¬ 
trol)  by  inspection,  thinking  a  lot,  debate, 
compromise,  and  seat-of-the-pants  intuition. 
Instead,  engineers  turn  to  laboratory  proto¬ 
types  and  computer  simulation  to  test  policies 
out  before  the  very  expensive  rocket  is  laun¬ 
ched.  Yet  we  expect  program  managers  to  manage 
weapon  system  procurement  involving  systems 
that  are  at  least  an  order  of  magnitute  more 


complex  than  purely  physical  systems,  and  to 
make  their  judgements  by  the  more  accustomed 
art  of  debate,  compromise,  etc.  The  results 
are  clear. 

It  is,  of  course,  unfair  to  say  that  program 
managers,  like  managers  of  most  social  systems, 
do  not  make  use  of  models.  Every  time  a  pro¬ 
gram  manager  looks  at  a  PERT  chart,  a  scale 
model  of  a  new  weapon  system,  or  a  blueprint 
of  a  new  system  component,  he  is  using  a  sub¬ 
stitute  for  the  real  system.  These  physical 
and  pictorial  models  are  similar  to  the  mathe¬ 
matical  models  used  in  computer  simulations  in 
that  they  provide  explicit  substitutes  for  the 
real  system.  Yet,  these  models  represent  physi¬ 
cal  subsystems  and  they  are  static.  Most  of 
the  interesting  (and  least  understood) 
aspects  of  weapon  procurement  systems  come  from 
the  dynamic  interactions  over  time  of  its  phy¬ 
sical  and  social  components.  PERT  charts, 
blueprints,  and  scale  models  cannot  help  mana¬ 
gers  (or  anyone  else)  project  the  system  be¬ 
havior  over  time  in  normal  circumstances,  let 
alone  under  the  influence  of  D&D's.  So  they 
turn  to  two  other  kinds  of  models.  In  most 
instances  managers  are  forced  to  turn  to  fuzzy, 
qualitative,  chameleon-like  mental  models  to 
serve  as  substitutes  for  the  real  system.  In 
a  growing  number  of  instances,  however,  managers 
are  trying  to  turn  to  explicit,  quantitative, 
mathematical  models.  These  models  span  the 
fields  of  econometrics,  operations  research, 
management  science,  and  applied  mathematics, 
and  have  been  used  to  study  physical,  biologi¬ 
cal,  and  social  systems.  Our  comments  that 
follow  report  on  one  specific  application, 
that  of  system  dynamics.  We  believe  this 
approach  to  be  general  and  that  it  can  be  ap¬ 
plied  to  any  system  acquisition. 

System  Dynamics 

System  dynamics  was  developed  during  the  1950' s 
by  Jay  W.  Forrester  and  his  colleagues  at  the 
Massachusetts  Institute  of  Technology  to  help 
in  their  study  of  industrial  systems  [l5]. 

Over  the  past  decade,  the  technique  has  been 
widely  applied  to  simulate  a  variety  of  social 
systems:  the  world  copper  market  [3],  natural 

resource  use  [.  5  ]  s  planning  and  control  for 
community  hospitals  [22],  decay  of  urban  sys¬ 
tems  [16],  the  U.S.  energy  system  [25 ] ,  regula¬ 
tion  of  electric  utilities  [l 4 ] »  and  logistics 
systems  [33]*  The  technique  is  most  widely 
known  for  its  use  in  the  study  of  The  Limits 
to  Growth  in  population  and  industrial  activity 
in  a  finite  world  [24]. 


The  heart  of  the  system  dynamics  paradigm  is 
the  view  that  social  systems  belong  to  a  gene¬ 
ral  class  of  nonlinear  feedback  systems.  To 
facilitate  the  representation  of  such  systems 
of  nonlinear  relationships,  the  system  dynamics 
technique  allows  for  easy  representation  of  non¬ 
linear^  relationships  and  chains  of  relationships 
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that  close  on  themselves  to  form  feec^ack  loops. 
It  is  the  interlocking  structure  of  multiple 
feedback  loops  that  the  system  dynamics  practi¬ 
tioner  seeks  to  find  the  explanation  of  the 
dynamic  behavior  of  complex  social  systems: 

It  is  in  the  positive  feedback  form  of 
system  structure  that  one  finds  the 
forces  of  growth.  It  is  the  negative 
feedback,  or  goal-seeking,  structure 
of  systems  that  one  finds  the  causes 
of  fluctuation  and  instability  [15]. 

To  build  a  system  dynamics  model,  one  follows  a 
series  of  steps  described  in  the  preface  to 
Dynamics  of  Growth  in  a  Finite  World  [23]  or 
those  discussed  in  Introduction  to  Urban  Dyna¬ 
mics  [2].  Generally,  the  following  steps  are 
required: 

t 

State  the  Problem  1- 

At  the  very  beginning  the  modeler  should  specify 


the  issues  the  model  is  to  address.  Usually, 
this  involves  a  description  of  a  particular 
system  that  is  behaving  ’’poorly”  and  a  question 
of  which  proposals  should  be  adopted  to  make 
the  system  behave  ’’better"  In  the  future.  In 
this  case  D&D's  are  the  issues  and,  for  example, 
the  setting  could  be  the  post-Milostone  HI 
(production)  phase  of  the  systems  acquisition 
process . 

Determine  the  Key  Factors  and  Their  Causal 
Interrelationships . 

In  this  step  the  modeler  lists  the  key  factors 
that  are  believed  to  cause  the  behavior  of 
concern.  The  interrelationships  among  these 
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Figure  7  AN  ILUJSTRA7I0N  OF  A  (NEGATIVE)  FEEDBACK  LOOP 

IN  THE  WEAPON  SYSTEM  ACQUISITION  CONTROL  PROCESS 
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factors  are  identified  and  portrayed  in  the 
form  of  causal  diagrams.  An  example  of  a 
causal  diagram  in  the  weapon  acquisition  sys¬ 
tem  is  given  in  Figure  7.  As  the  numerous 
interrelationships  in  any  system  are  identi¬ 
fied,  some  series  of  relationships  form  a 
closed  chain  or  feedback  loco.  In  Figure  7, 
such  a  loop  is  noted  by  a  Q  .  It  is  in 
the  interaction  of  these  loops  that  the 
modelers  seek  to  explain  the  bahavior  of  the 
system  under  study. 

Formulate  Model  in  Flow  Diagram 

In  this  step  the  representation  of  the  system 
is  reformulated  in  the  form  of  a  flow  dia¬ 
gram  which  facilitates  the  eventual  repre¬ 
sentation  in  the  form  of  a  set  of  difference 
equations.  The  flow  diagram  is  especially 
helpful  in  distinguishing  between  flows  of 
material  and  flows  of  information.  Morever, 
the  flow  diagram  forces  the  modeler  to 
indicate  explicitly  any  delays  or  nonlinear 
relationships  included  in  the  model. 

The  use  of  causal  loop  diagrams  and  flow 
diagrams  not  only  helps  the  modeler  to  keep 
track  of  his  progress,  but  it  helps  com¬ 
municate  the  important  features  of  the  model 
to  potential  users  as  well.  The  use  of 
meaningful  variable  names--names  that  can  be 
seen ,  "  fel  t" ,  or  talked  about--together  with 
the  diagraming  aids,  gives  properly  construc¬ 
ted  system  dynamics  models  a  good  chance  of 
bridging  the  communication  barrier  between 
the  model  builder  and  the  model  user.  This 
barrier  has  been  described  by  Goldie  as 
follows: 

When  we  attempt  to  use  the  new  techniques 
that  management  scientists  advocate,  we 
suddenly  find  that  we  are  out  of  the  loop. 

A  bright  young  man  takes  my  problem  away 
and  translates  it  from  managerese  to 
computerese  [17]. 

In  speaking  about  this  opportunity  for 
bridging  the  communication  barrier,  users 
ofter  refer  to  the  system  dynamics  approach 
as  "common  sense  quantified,"  whereas  more 
elaborate  techniques  are  described  as  "com¬ 
mon  sense  made  difficult." 

Estimate  the  Parameters  of  the  Model 

In  this  step  the  parameter  values  must  be 
estimated.  Each  of  the  parameters  of  a 
system  dynamics  model  is  estimated  individu¬ 
ally  from  the  best  information  available. 

This  information  can  take  the  form  of  highly 
accurate  physical  measurements  and  make  use 
of  volumes  of  hard  statistical  data  (at  one 
end  of  the  spectrum)  or  expert  opinion  and 
individual  intuition  (at  the  other).  The  use 
of  expert  opinion  in  the  absence  of  more 
formal  data  to  help  parameter  identification 


tends  to  make  some  data  purists  squirm.  Yet, 
to  rule  out  subjective  inputs  if  they  are 
the  only  source  of  information  available 
ignores  the  fact  that  most  of  the  relevant 
data  in  any  real  social  system  is  contained 
in  the  heads  of  people  and  is  not  easily 
recorded  in  the  form  of  time  series  or  cross 
sectional  data. 

Practitioners  of  other  modeling  techniques 
prefer  to  ignore  relationships  for  which 
hard  statistical  data  is  not  available. 
Leaving  such  relationships  out,  however, 
results  in  assigning  them  the  parameter 
value  that  most  people  would  agree  is  wronq- 
-zero . 

By  incorporating  subjective  judgments  in 
estimating  parameter  values,  the  system 
dynamics  practitioner  can  determine  through 
sensitivity  testing  those  areas  where  further 
debate  and  data  collection  is  warranted.  If 
a  model's  behavior  is  generally  insensitive 
to  certain  parameter  values,  scarce  resources 
should  not  be  wasted  in  collecting  data  to 
estimate  them. 

Generate  Initial  Output  and  Increase 
Confidence  in  the  Model 

Once  parameter  values  have  been  estimated, 
DYNAMO  equations  of  a  system  dynamics  model 
can  be  easily  constructed.  (DYNAMO  is  a 
software  package  usually  used  to  construct 
and  test  system  dynamics  models.)  If  the 
initial  output  generated  by  the  model  fails 
to  imitate  the  real  system’s  behavior  well 
enough,  the  modeler  returns  to  Step  2  and 
begins  an  interactive  process.  We  should 
emphasize  that  system  dynamics  models  do  not 
automatically  reproduce  historical  trends 
since  model  parameters  are  estimated 
individual! y.  This  procedure  is  quite 
different  from  modeling  approaches  wherein 
model  parameters  are  estimated  "all  at  once" 
to  give  the  best  fit  to  historical  trends 
over  a  certain  period  of  time.  Thus,  it 
should  come  as  no  surprise  that  such  models 
are  capable  of  providing  extremely  close 
fits  to  historical  trends  over  the  time 
period  used  in  the  parameter  estimation. 

They  are  designed  to! 

Generating  a  reasonable  pattern  of  behavior  - 
both  over  some  historical  period  and  into  the 
future  -  is  but  one  of  several  tests  that 
should  be  performed  to  increase  confidence 
in  a  model.  Sensitivity  testing  should  be 
performed  to  ensure  that  the  model  does  not 
respond  unreal istically  to  changes  in  par¬ 
ticular  input  values.  The  response  of  a 
model  to  disturbances  in  exogenous  inputs 
should  be  examined  for  plausibility.  These 
and  other  confidence  tests  are  described  in 
Forrester's  Industrial  Dynamics  115  ]. 
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Simulate  the  Effects  of  Proposed  Changes 

In  this  final  step  the  modeler  simulates 
the  effects  of  pending  or  actual  changes  and 
observes  change-induced  D&D's  to  determine 
the  explicit  effect  on  the  system  schedules, 
costs,  and  performance. 

For  example,  there  are  at  least  two  kinds  of 
well-known  D&D's  that  can  effect  system 
behavior.  First,  there  are  changes  that 
are  under  the  control  of  the  program  manager. 
For  example,  a  program  manager  may  be 
considering  the  merits  of  giving  a  go-ahead 
on  stretching  the  production  schedule  vis- 
a-vis-  simply  slipping  the  entire  schedule 
by  a  uniform  amount.  A  second  kind  of 
change  is  one  imposed  on  the  system  from 
the  outside.  For  example,  a  large  cost  of 
living  wage  increase  for  production  workers, 
a  rise  in  the  cost  of  material  used  In 
construction,  or  congressional  changes  in 
the  authorized  budget  for  the  program. 

In  these  instances  there  are  two  questions 
to  be  asked:  (1)  What  is  the  effect  of 
the  change  on  the  system  if  the  program 
manager  does  nothing?  and  (2)  What  action 
can  the  program  manager  take  in  the  face 
of  this  change  to  improve  system  response 
to  it?  In  Step  6  each  kind  of  change,  as 
well  as  combinations  of  changes,  can  be 
and  have  been  examined. 

THE  INGALLS  SHIPBUILDING  MODEL: 

A  CASE  STUDY 

A  recent  study  performed  by  Pugh-Roberts 
Associates  involved  the  application  of 
system  dynamics  to  a  dispute  between  the 
United  States  Navy  and  Ingalls  Shipbuilding 
in  Pascagoula,  Mississippi,  a  division  of 
Litton  Industries,  Inc.  [11,  12  \  This 
study  is  reported  in  detail  by  Cooper  (1980). 
The  dispute  focused  on  issues  of  delays  and 
design  changes  and  the  long-range  higher- 
order  impacts  of  design  changes.  In  1969 
Ingalls  was  awarded  a  firm-fixed-price 
contract  by  the  Navy  to  design  and  build 
nine  (subsequently  reduced  to  five)  am¬ 
phibious  assault  ships  (LHA's).  As  repor¬ 
ted  by  Cooper,  the  LHA  is  "20  stories  high 
and  the  length  of  three  football  fields." 

It  is,  according  to  the  Navy,  "the  largest, 
fastest,  and  most  versatile  vessel  in  the 
history  of  American  amphibious  war  fare." 

The  Navy  contracted  with  Ingalls  for  the 
LHA  program  as  one  of  the  two  Navy  ship¬ 
building  programs  using  a  Total  Package 
Procurement  approach.  This  meant  that 
Ingalls  was  provided  only  with  performance 
specifications  and  was  thereafter  "solely 
responsible  for  all  system  design,  detailed 
design;  material  procurement,  planning, 
testing,  and  construction."  The  other 
program  was  the  30-ship  Destroyer  DD963 
program,  also  awarded  to  Ingalls  in  mid  1970. 


Cooper  sets  the  stage  for  an  inevitable  D&D 
problem  as  follows: 

The  design  and  construction  of  the  new, 
complex  LHA's  and  the  DD963  Spruance- 
class  destroyers  (twice  the  size  of 
destroyers  of  the  prior  generation) 
dominated  the  operations  at  Ingalls 
throughout  the  1970'$.  Either  of  the 
LHA  or  DD963  programs  would  have  re¬ 
quired  a  significant  facilities  and 
manpower  expansion  for  any  shipyard  - 
Ingalls  more  than  doubled  its  workforce 
for  the  two  programs.  During  this  time 
there  were  periodic  nationwide  material 
shortages  and  a  critical  scarcity  of 
skilled  shipbuilding  labor.  A  new  form 
of  organization  for  ship  design  was 
being  used  by  Ingalls  and  the  Navy. 

Formal  requirements  were  instituted 
for  integrating  with  the  usual  design 
effort  the  consideration  of  vessel 
maintainability,  reliability,  logis¬ 
tics  support,  manning,  and  more. 

It  was  in  this  setting  that  thousands 
of  design  changes  on  the  LHA  were 
received  by  Ingalls  from  the  Navy  [11] . 

In  January  1976  Ingalls  approached  Pugh- 
Roberts  Associates,  convinced  that  the 
major  contributing  factor  to  overruns  of 
some  $500  million  was  the  disruptions  that 
had  been  present  in  every  phase  of  the  LHA 
program.  Ingalls  further  felt  that  "... 
Navy-responsible  delays  and  design  changes 
had  contributed  to  the  disruption  and  had 
affected  many  areas  which  otherwise  would 
not  have  experienced  difficulties."  (Of 
course,  the  counter  to  placing  all  the 
blame  for  cost  overruns  on  the  Navy  are 
charges  that  the  contractor  "bought-in"  or 
contractor  mismanagement.)  Cooper  points 
out  that: 

Claims  against  the  Navy  have  evolved 
to  a  form  in  which  two  major  segments 
can  be  identified.  First,  the  direct 
impact,  or  the  "hard-core"  costs,  of 
a  design  change  or  delay  are  estimated. 
While  there  are  always  legal  questions 
of  entitlement,  these  costs  are  not 
difficult  to  understand  and  can  be 
quanti fi ed--for  example,  the  number  of 
man-hours  required  to  effect  the  change 
in  a  design  drawing  and  the  man-hours 
needed  to  implement  the  immediate 
change  in  the  construction  of  the  ship. 
The  second  segment  of  the  claims 
consists  of  "delay  and  disruption" 
costs-- the  second  and  third-order 
"ripple  effects"  of  dealing  with  the 
direct  changes.  (These  are  tradition¬ 
ally  the  most  difficult  issues  to 
quantify  and  justify.)  In  concept, 
they  are  the  "snowballing"  effects 
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within  a  work  phase,  among  work  phases, 
and  between  work  programs,  such  as 
altered  work  sequence,  conflicting 
facilities  and  manpower  requirements, 
skill  dilution,  undetected  work  errors, 
and  more.  The  ultimate  consequences 
for  program  performance  include  the 
additional  cost  and  time  required 
to  accommodate  the  full  range  of 
effects  of  the  direct  changes!  11 J. 

As  pointed  out  by  Cooper,  changed  designs 
produce  a  need  to  "rip  out  and  rework," 
which  means  increased  manpower  and  material 
costs,  and  manpower  diverted  from  other 
programs  causing  schedule  changes,  reduced 
producti vity,  and  so  on.  All  these  "rip¬ 
ple"  effects  work  together  and  "feed  upon 
one  another  in  vicious  circles  that  continue 
to  exaggerate  the  cost  and  schedule  impacts 
of  design  changes  far  beyond  the  time  and 
stage  of  work  directly  affected.  This  is 
the  essence  of  'delay  and  disruption [11  1 

The  methodology  used  to  tackle  these  D&D- 
induced  claims  by  Ingalls  was  system  dynamics. 
It  was  a  methodology  that  would  "(a)  cor¬ 
rectly  quantify  Navy-responsible  delay  and 
disruption  costs  in  the  design,  procurement, 
planning,  and  production  stages  of  the 
programs,  and  (b)  demonstrate  the  cause- 
effect  relation  of  the  costs  to  the  items 
cited  in  the  'hard-core'  segment  of  the 
claim."  Since  a  model  was  developed  and 
presented  by  Cooper,  only  essential  parts 
are  repeated  here.  The  model  developed  by 
Cooper  was  "...  a  replica  of  the  management 
decisions  and  operations  of  the  company. 

The  single,  important,  intended  use  was 
qualification  of  the  comprehensive  impacts  of 
customer  changes  and  delays." [11]  The  quanti¬ 
fication  of  impacts  was  accomplished  by  run¬ 
ning  the  simulation  model  twice. 


In  one  run,  the  model  calculated  the  number 
and  timing  of  man-years  expended  in  each 
program  phase  from  the  beginning  of  the 
programs  to  completion  and  all  of  the  Navy- 
responsible  changes  cited  in  the  hard-core 
claim  items  were  i ncl uded .  This  run  re¬ 
created  and  forecast  the  actual  work 
schedules  and  expenditure  of  man-years  on 
the  LHA  and  DD  programs,  including  all  the 
"ripple,  D&D,  effects."  . 

The  second  run  was  identical  to  the  first 
except  al  1  imputs  that  represented  the  Navy- 
responsible  changes  cited  in  the  hard-core 
claims  were  removed.  The  results  of  this 
simulation  run  were  the  estimated  schedules 
and  man-hours  which  would  have  occurred  had 
the  Navy  not  intervened  in  the  LHA  program. 

The  differences  between  the  two  runs  were 
the  quantification  and  diagnosis  of  the  cost 


impacts  caused  by  the  Navy  changes  and 
submitted  to  the  government  in  October  1977. 
It  was  the  sole  technical  basis  for  the 
majority  of  the  claim. 

In  the  settlement  reached  in  June  1978, 
INgalls  received  $447  million  from  the 
Navy.  It  was  the  first  time  that  the 
Navy  had  given  such  a  substantial  consid¬ 
eration  to  a  delay  and  di  sruption  claim. 

The  managers  and  lawyers'  estimates  place 
the  model's  dollar  contribution  to  the 
settlement  between  $170  and  $350  million. 

The  model  also  eliminated  the  adversary 
relationship  between  Ingalls  and  its  best 
customer--the  Navy.  Since  the  avoidance 
of  contractor  claims  against  the  govern¬ 
ment  was  cited  as  a  hi gh-priori ty  ob¬ 
jective  in  the  procurement  process  study 
prepared  under  the  Secretary  of  the  Navy, 
and  since  change-induced  D&D's  seem  to  be 
inevitable  in  complex  defense  systems 
acquisition,  it  would  seem  that  system 
dynamics  provides  a  methodology  and  a 
technique  for  compreshnsi ve  analysis  and 
a  mechanism  for  managing  D&D's  on  the 
part  of  both  the  defense  contractor(s) 
and  the  defense  program  managers.  The 
question  remains  one  of  Implementing  and 
intitutional izing  such  a  management 
technique. 

IMPLEMENTING  AND  INSTITUTIONALIZING 

D&D  MANAGEMENT 

Although  the  D&D  concept  developed  in  this 
paper  is  somewhat  broader  than  in  the 
specific  Navy-related  case  study  presented, 
the  message  is  the  same.  Once  a  contract 
is  awarded,  a  basic  tool  for  the  management 
of  the  acquisition  should  be  a  model  that 
can  be  used  to  determine  causes,  effects 
(including  costs),  and  who  pays  for  the 
impacts  of  change-induced  D&D's. 

The  model  would  be  developed  to  specifi¬ 
cally  link  changes  to  their  D&D's  and 
D&D's  to  their  impacts  on  cost,  per¬ 
formance,  and  schedule.  As  such,  it 
would  become  the  primary  management  mech¬ 
anism  to  determine  the  cost-benefits 
(or  cost-effectiveness)  of  proposed  design 
changes  (for  example).  It  would  be  the 
primary  management  mechanism  to  provide 
for  dollar  adjustments  in  funding  levels 
or  to  provide  for  schedule  revisions  in 
the  1  ight  of  changes . 

Implementing  and  institutionalizing  such 
a  methodology  for  management  use  and  phil¬ 
osophical  acceptance  will  not  be  easy. 

And  here  we  make  a  distinction  between 
implementation  and  Institutionalization. 

It  is  relatively  easy  to  implement  a  change 
in  procedures,  policies,  or  management 


* 


218 


approach.  To  see  that  the  new  approach  re¬ 
mains  implemented  over  some  lengthy  time 
horizon  (institutionalization)  is  quite 
another  matter.  There  are  many  examples 
within  DgO  of  approaches  (see  Folkways 
section)  that  have  been  implemented,  but 
that  have  failed  to  become  institutionalized. 

It  seems  to  the  authors  that  a  key  ingredient 
to  both  implementation  and  institutionali¬ 
zation  is  the  active  interest  and  support 
of  both  the  top  management  and  working 
management  levels  in  both  the  D  D  and 
Industry.  Such  support  is  typically  earned 
in  many  ways,  but  there  are  three  necessary 
(if  not  sufficient)  prerequi si tes :  (1) 

The  approach  proposed  must  be  "effective." 

It  must  work  and  yield  tangible  and  readily 
recognizable  results  and,  (2)  it  must  also 
be  "efficient"  and  not  cost  more  than  its 
measurable  worth  at  a  minimum  and,  (3) 
it  must  be  understood  by  those  who  are  to 
support  and  use  it. 

The  Ingalls  case  study  seems  to  go  a  long 
way  toward  meeting  the  first  two  prerequi¬ 
sites.  We  are  not  as  optimistic  that 
the  third  has  as  yet  been  tackled  on  any 
significant  scale.  We  suggest  that 
education  (special  training  programs,  short 
courses,  intensive  seminars  such  as  this 
one,  master's  degrees,  etc.),  can  help 
provide  the  conceptual  understanding  and 
develop  skill  through  practice,  and 
familiarity  to  meet  the  thi  rd  prerequi si  ter 

The  authors  are  not  aware  of  any  large 
scale  effort  to  provide  such  educational 
or  training  opportunities.  Where  simu¬ 
lation  is  formally  taught  at  all  (and 
even  then  the  scale  and  number  of  trainees 
is  relatively  small),  the  techniques  of 
simulation  seem  to  dominate  if  not  get  in 
the  way  of  applications. 

System  dynamics,  which  we  feel  is  particu¬ 
larly  well  suited  to  promote  understanding 
among  users  with  even  modest  technical 
skills  in  computers  and  simulation,  is 
taught  at  relatively  few  institutions 
around  the  country.  We  are  not  aware  of 
any  major  effort  to  train  or  develop  program 
managers  engaged  in  the  systems  acquisition 
process  on  techniques  such  as  this. 

Implementation  could  occur  in  the  form  of 
DOD  directives  and/or  MIL  Standards.  The 
directives  or  standards  would  define,  at 
least  to  some  degree,  the  appropriate 
ingredients  of  a  model  to  be  developed 
for  managing  D&D's  for  any  given  acquisition. 
They  might,  but  probably  should  not,  specify 
the  detailed  variables  and  causal  loop 
diagrams  to  be  flow  diagramed  and  parameter¬ 
ized.  They  might,  and  perhaps  should, 
specify  the  boundaries  beyond  which  no  model 


should  extend. 

The  main  initial  activity  and  function  of  a 
program  manager  within  the  DOD  would  be  to 
speci fy  measurement  methods  for  each  vari¬ 
able,  parameter,  table  function,  or  other 
quantifiable  component  of  the  simulation 
model.  They  would  then  spend  enormous 
amounts  of  time  and  effort  working  with  the 
defense  contractor,  refining  and  validating 
the  model  so  that  both  agree  that  it  is  a 
fair  representation  of  the  real  world  with 
respect  to  the  D&D  issues.  Then,  as  design 
changes  and  subsequent  other  D&D  occur,  their 
impacts  would  be  traced  through  their 
logical  impacts  on  costs,  schedules,  and 
system  performance. 

For  those  impacts  that  can  be  traced  to 
government-generated  D&D's,  the  government 
would  pay  (as  with  an  approved  ECP).  For 
those  D&D's  that  are  contractor  related, 
the  contractor  should  absorb  them. 

For  D&D's  that  share  a  common  responsi  bil  ity 
or  are  the  result  of  "mother  nature," 
suitable  compensation  or  settlement  rules 
could  be  devised.  Trade-offs  could  be 
made,  using  the  model,  with  the  government's 
program  manager  to  report  on  impacts  and 
the  cost-effectiveness  of  proposed  changes. 

The  defense  contractor  could  use  the  same 
model  to  allocate  resources  in  an  "optimal" 
way  in  the  light  of  D&D's  or  claims  could 
be  reviewed  or  investigated  and  resolved 
without  the  time  consuming  and  expensive 
(both  in  costs  ana  attitudes)  process  of 
1  i  tigation . 

Once  agreed-on  models  had  been  hammered  out, 
most  case-by-case  D&D's  could  "automatically" 
be  managed.  Only  in  response  to  changed  cir¬ 
cumstances  in  the  real  world  would  the  model 
be  altered  to  conform  once  again  with  the 
real  world.  The  model  would,  within  the 
acquisition  circle  rel  event  to  the  acqui¬ 
sition,  be  public  knowledge.  This  metho¬ 
dology  for  dealing  with  risk  and  uncertainty 
could  be  used  as  an  on-going  management 
tool  and  not  only  for  D&D  management. 

In  short,  this  approach  to  D&D  management 
allows  D&D  conflicts  to  be  objectively 
discussed,  reviewed,  and  resolved  at  the 
systems  level  rather  than  on  a  case-by-case 
basis  for  each  D&D  in  the  press,  the  Con¬ 
gress,  or  in  the  courts.  Such  a  model,  once 
specified,  should  define  and  remove  nearly 
all  ambiguity  without  impairing  the  program 
manager  or  the  defense  contractor's  ability 
to  manage. 

Potential  saving  in  financial  and  social 
costs,  delays,  frustrations,  etc.,  while 
believed  to  be  significant,  may  be  fully 
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incalculable.  We  consider  the  idea  of 
managing  D&D's  at  the  system  level  by  a  system 
dynamics  simulation  model  to  have  the  poten¬ 
tial  of  revol  utionary  impact  on  program 
management. 

Any  broad  application  of  a  new  conceptual 
approach  is  bound  to  be  full  of  unexpected 
consequences,  booby  traps,  and  surprises  for 
the  untrained.  For  a  while,  therefore,  the 
wise  impl  ementor-to-be  would  want  to  run  some 
tests  (develop  three  or  four  small  program 
models)  to  fully  develop  the  approach  before 
implementing  it  DoD-wide. 

We  therefore  suggest  that  an  experimental 
test  of  this  approach  be  developed  and  im¬ 
plemented  using  carefully  selected  acquisi¬ 
tions  (program)  as  test  beds  for  the  methodol¬ 
ogy.  The  results  of  such  tests  could  provide 
the  necessary  insight  and  feedback  to  permit 
a  determination  of  the  cost-effectiveness  of 
such  a  management  tool  in  the  management  of 
D&D's. 

We  know  of  no  such  test  setting  in  which  even 
limited  experimentation  with  the  method  we 
advocate  is  occurring  at  government  request. 
But  we  have  hopes. 
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METHODOLOGIES  PANELS  SUMMARY 

This  session  of  the  workshop  concentrated 
on  the  area  of  "Risk  and  Uncertainty  Methodo¬ 
logies."  The  papers  delivered  by  the  speak¬ 
ers  and  the  intense  interaction  of  panelists, 
achieved  the  purposes  outlined  for  this 
section. 

The  panel  membera  described  a  number  of 
methodologies  and  techniques  which,  in  spite 
of  the  problems  of  definition  and  the  need 
for  more  refinements,  are  considered  of  prac¬ 
tical  importance  and  should  be  applied.  The 
range  and  thrust  of  the  recommendations  from 
these  panels  are  an  indication  of  the  momen¬ 
tum  that  was  attained.  The  seriousneaa  re¬ 
garding  research  was  also  evident  from  the 
recommendations  for  future  activities. 

The  first  objective  was  to  identify  and 
clarify ^ existing  methodologies  for  a  large 
project  s  life  cycle.  One  methodology  was 
presented  using  entropy,  a  concept  to  develop 
a  model  that  would  assist  project  managers  in 
predicting  the  final  program  cost  when  un¬ 
certainty  is  present.  The  methodology  was 
used  in  two  projects  to  validate  the  approach. 
The  model,  although  static  in  nature,  show¬ 
ed  mixed  results.  The  methodology  illus¬ 
trated  that  formal  links  between  entropic 
models  and  methods  for  determining  informa¬ 
tion  requirements  from  limited  data  can  be 
established. 

Another  report  reviewed  research  on 
network  models  for  project  management  under 
conditions  of  risk.  There  were  several  other 
techniques  for  assessing  subjective  proba¬ 
bilities  discussed.  These  included  the: 
Choice-Between-Gamb les ,  Standard  Lottery, 
Cnurchman-Ackof f ,  Delphi,  DeGood  Consensus, 
and  Direct  Estimation. 

Possibilistic  uncertainty  was  the  sub¬ 
ject  of  another  presentation  and  panel  dis¬ 
cussion.  The  mathematical  development  of 
"fuzzy  sets"  theory  was  presented  as  a  method¬ 
ology  for  the  analyst  and  decision  maker  to 
provide  results  which  more  closely  represent 
the  actual  meaning  of  the  criteria  that  are 
to  be  satisfied. 

The  major  approach  of  another  panel  was 
to  discuas  the  use  of  moments  in  coat  riak 
analysis  (Wilder  and  Black).  The  presenta¬ 
tion  described  a  methodology  and  procedure  for 
assessing,  and  by  implication,  to  some  degree 
managing,  cost  risk  during  the  preliminary 
product  design  stage  in  the  life  cycle.  Ex¬ 
pert  judgments  were  used  to  obtain  most  like¬ 
ly  values.  These  in  turn  were  treated  as 
nodes,  with  upper  and  lower  bounds,  and  used 
to  form  a  triangular  distribution. 

The  process  of  aggregating  uncertainties 


for  each  element,  which  is  the  heart  of  the 
procedure,  uses  the  method  of  moments.  Ap¬ 
propriate  adaptations  of  this  closed-form  pro¬ 
cedure  exist  for  various  rules  for  combining 
elements  into  the  aggregate  system.  The  out¬ 
put  is  expressed  by  "fitting"  the  final  mo¬ 
ment  outputs  to  a  beta  (or  occasionally  other) 
distribution.  The  properties  of  that  beta 
distribution  are  then  expressed  in  the  form 
of  a  cumulative  distribution  function  showing 
the  probability  that  the  system  will  be  com¬ 
pleted  at  each  of  a  range  of  possible  costs. 

System  dynamics  was  the  methodology  pre¬ 
sented  and  reviewed  in  another  panel  (Sherman 
and  Gardiner).  The  methodology  presented  co¬ 
vered  a  computerized  simulation  model  (System 
Dynamics)  used  to  reconstruct  a  major  ship¬ 
building  program  that  attempted  to  resolve  a 
major  litigation  between  a  contractor  and  the 
government.  The  model  achieved  the  objectives 
of  quantifying  the  amount  of  delay  and  dis¬ 
ruption  costs  in  the  design,  procurement,  plan- 
ning ,  and  production  stages  of  the  program. 

The  model  also  demonstrated  the  cause-effect 
relationship  of  costs  and  identified  responsi¬ 
bility  for  the  delays  and  disruptions  that 
generated  the  time/cost  overruns.  Risk  and 
uncertainty  were  reviewed  in  the  context  of 
disruption  in  systems  acquisition.  The  as¬ 
sumption  was  that  risk  and  uncertainty  are  in¬ 
herent  in  any  new  project  and  therefore  dis¬ 
ruption  will  occur.  The  emphasis  should  be 
placed  on  the  predication  of  disruption  and 
the  uae  by  management  of  this  information. 

Another  task  of  the  panel  was  to  assess 
the  strengths  and  application  of  the  various 
methodologies.  A  summary  of  comments  follows: 

1*  Entropy  can  potentially  be  used  as  an  in¬ 
dication  of  disorder  in  information  re¬ 
lated  to  key  variables  in  a  program. 

2.  Six  techniques  were  reviewed  for  assess¬ 
ing  subjective  probabilities.  While  no 
one  best  way  surfaced,  the  Delphi  ap¬ 
proach  emerged  as  the  one  most  accept¬ 
able. 

3.  Fuzzy  sets  are  useful  in  multicriterion 
decision  theory.  They  permit  trade-offs, 
and  analysis  of  alternative  approaches 
with  multiple  objectives. 

4.  Fuzzy  sets  permit  the  use  of  soft  versus 
hard  constraints  and  are  useful  for  vis¬ 
ualizing  imprecise  (impossibilis tic)  un¬ 
certainty)  representations. 

5.  Fuzzy  sets  theory  is  presently  used  in 
artificial  intelligence  models,  build¬ 
ing  multiattribute  models,  and  for  fore¬ 
casting. 

6.  The  use  of  moments  in  cost  risk  analysis, 
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shows  ways  of  relaxing  the  assumption  of 
additivity  of  couts.  The  method  may  be 
used,  for  example,  to  deal  with  time  risks 
which  may  well  not  be  additive. 

7 .  The  advantage  of  using  the  method  of 
moments  over  such  alternatives  as  Monte 
Carlo  sampling  is  that  it  requires  less 
computer  time  and  is  easier  to  understand 
the  results.  The  method  also  produces  a 
clearer  picture  of  which  elements  of  the 
total  system  development  process  will  in 
fact  be  the  cost  drivers,  both  for  the  mean 
cost  and  for  its  variance.  This  approach 
helps  to  focus  attention  on  the  most  im¬ 
portant  issues. 

8.  Simulation  and  system  dynamics,  in  par¬ 
ticular,  facilitate  modeling  of  complex 
systems  and  permit  Mwhat  if”  analysis.  A 
significant  advantage  of  simulation  is 
its  predictive  value  to  project  managers. 

9.  A  system  dynamics  model  can  predict  the 
impacts  of  disruption  throughout  the  phases 
of  a  project's  life  cycle. 

10.  Because  system  dynamics  is  a  dynamic  metho¬ 
dology,  it  can  contribute  more  than  a  typi¬ 
cal  static  analysis  to  understanding  chan¬ 
ges  over  time. 

11.  Fuzzy  sets  allow  a  robust  representation 
because  of  the  ability  to  "grade"  the 
characteristics.  For  example,  there  are 
gradations  of  acceptability  ranging  to 
unacceptable  rather  than  the  simple  thre¬ 
shold  used  in  crisp  sets, 

12.  Fuzzy  sets  formulation  allows  the  normal 
"goodness"  words  to  be  readily  converted 
into  mathematical  expressions. 

13.  A  probabilistic  network  model  which  permits 
branching  at  nodes,  can  consider  the  three 
variables  of  time,  cost  and  performance. 

It  also  includes  a  wider  choice  of  built- 
in  probability  functions  .(constant,  uni¬ 
form,  triangular,  normal,  lognormal, 
Poisson,  Erlang,  gamma,  beta,  and  tabu¬ 
lated  distributions). 

1A.  The  interactive  version  of  the  RISCA  model 
features  fullypromp ted  initialization  and 
modification  of  the  networks  from  keyboard 
to  system  printer. 

Some  of  the  weaknesses  noted  in  the  application 

of  methodologies  discussed  were: 

1 .  The  entropic  model  (Martin)  needs  further 
validation. 

2.  The  use  of  the  method  of  moments  does  not 
examine  ways  of  relaxing  the  assumption 
of  independence  among  elements. 


A  methodological  problem  that  was  discus¬ 
sed  concerned  the  use  of  moments  and  the 
interaction  between  estimation  of  cost  re¬ 
lationships,  errors,  and  for  costs  assessed 
by  techniques  such  as  regression  analysis. 
This  was  seen  as  a  subtle  but  not  unmanag- 
able  issue. 

5.  Another  issue  concerning  the  use  of  moments 
was  the  problem  produced  by  the  combination 
of  the  Central  Limit  Theorem  and  the  inde¬ 
pendence  assumptions.  If  the  number  of 
probability  distributions  convolved  to 
obtain  total  system  cost  distributions,  is 
large,  regardless  of  the  form  and  character 
of  the  imput  distributions,  the  output 
distribution  will  be  relatively  normal,  with 
relatively  small  variance  and  little  or  no 
skewness.  While  no  solution  is  suggested, 
most  panelists  agreed  that  outputs  with 
small  variance  and  no  skewness  are  unreal¬ 
istic  discriptions  of  the  results  expected. 

6.  A  discussion  of  sensitivity  analyses  led  to 
the  conclusion  that,  while  indispensable  to 
analysts,  they  were  probably  confusing  to 
decision  makers,  and  should  usually  be  held 
in  reserve  unless  they  led  to  very  clear 
conclusions  not  visible  in  the  original 
analysis. 

7.  A  weakness  of  the  use  of  system  dynamics  is 
inherent  in  any  large  computerized  simula¬ 
tion,  namely  the  development  of  an  accurate 
model  including  the  time  and  cost  functions 
involved . 

8.  Emphasis  should  be  placed  on  the  importance 
of  attaching  caveats  of  various  kinds  to 
estimates,  especially  when  those  estimates 
depend  in  any  important  way  on  independence 
assumptions . 

The  panels  were  asked  to  summarize  the  state-of- 
the-art  on  methodologies  for  risk  and  uncertainty 
analysis.  The  advances  have  been  dramatic. 

They  range  from  increased  mathematical  applica¬ 
tions,  such  as  fuzzy  sets,  to  a  greatly  expanded 
use  of  the  computer,  not  only  for  analysis,  but 
for  complex  system  model  building.  Most  parti¬ 
cipants  agreed  that  currently  there  were  inade¬ 
quate  ways  to  communicate  these  state-of-the- 
art  advances  in  risk  and  uncertainty  analysis  to 
the  parties  and  organizations  (stakeholders) 
concerned.  This  perceived  problem  lead  to  the 
final  objective  presented  to  the  panalists. 

RECOMMENDATIONS 

The  panel  members  were  asked  to  suggest  ways  of 
integrating  the  use  of  methodological  approaches 
into  the  DoD  project  acquisition  process.  The 
recommendations  included  (not  in  order  of  im¬ 
portance)  the  following: 

1-  Develop  formal  means  for  relating  the  en¬ 
tropic  model  to  the  methods  for  determining 


m 


t 


maximum  information  from  limited  data. 

2.  Identify  the  hierarchy  of  activities,  and 
number  of  activities  with  potential  im¬ 
pact  on  uncertainty. 

3.  Extend  the  validation  of  the  Martin  model 

(and  possibly  develop  an  improved  model) 
and  test  the  model  in  the  following 
areas:  (a)  psychology,  (b)  construc¬ 

tion,  (c)  portfolio  management. 

4.  Develop  a  revised,  validated  Martin  model 
so  that  it  can  become  a  useful  manager¬ 
ial  tool. 

5.  Expand  the  application  of  the  fuzzy  set 
theory  to  allow  greater  use  of  expert, 
algorithms  and  questionaires  for  build¬ 
ing  evolutionary  aystems,  and  querying 
data  bases. 

6.  A  pilot  application  of  fuzzy  sets  theory 
should  be  performed  to  evaluate  a  pre¬ 
ferred  technical  approach  in  the  devel¬ 
opment  of  an  acquisition  strategy. 

The  results  of  that  study  be  presented 
at  the  next  conference. 

7.  Use  the  method  of  moments  in  cost  analy¬ 
sis  to  compare  two  DoD  programs,  one 
staffed  with  trained  cost  analysts, 

the  other  not. 

8.  Conduct  research  on  the  concept  of  orga¬ 
nizational  slack. 

9.  Develop  a  real  world  application  of 
system  dynamics  on  a  new  project  or 
system  for  predicting  disruption  and 
assigning  responsibility  (using  an  ECP 
type  arrangement) ,  as  a  technique  for 
the  management  of  disruption. 

10.  Select  a  new  system  or  project  and 

run  a  parallel  development  (systems  mo¬ 
del  at  milestone  0),  working  with  a  con¬ 
tractor  and  the  project  manager  on  the 
model.  This  would  serve  as  an  example 
to  validate  the  use  of  system  dynamics. 
The  purpose  would  be  to  test  and  validate 
the  methodology  in  a  real  application. 

11.  Expand  the  Cochran  disruption  model  and 
conduct  research  to  verify  the  theoreti¬ 
cal  concepts, 

12.  Generate  a  glossary  of  terminology  to  be 
added  to  the  taxonomy. 

13.  Develop  training  programs  targeted  spe¬ 
cifically  at  project  managers  using 
system  dynamics. 

14.  Consider  renaming  our  area  of  interest 
from  risk  and  uncertainty  analysis  to 
the  management  of  disruption  or  the 


management  of  risk  and  uncertainty. 

15.  Establish  a  national  center  for  the 
study  of  distuption  and  elicit  DoD  or 
service  support  for  the  establishment 
of  a  designated  national  data  bank  for 
the  retention  of  information  on  the 
study  of  risk  and  uncertainty  methodology. 

16.  Create  a  central  repository  of  metho¬ 
dologies  (from  above)  catalogued  using 
the  taxonomy  developed  for  ready  ref¬ 
erence  with  explanationa ,  working  ex¬ 
amples  and  an  assessment  of  usefulness 
the  methodologies. 

17.  Conduct  research  on  the  relationship 
between  management  behavior  and  con¬ 
tract  type. 

18.  Concentrate  on  the  management  of  dis¬ 
ruption  rather  than  trying  to  solve 
overrun  problems  by  devision  new  types 
of  procurement  contracts. 

19.  Use  dynamic,  nonlinear,  time  indexed, 
Simula taneous  equations  for  simulation 
analysis . 

20.  Conduct  a  survey  of  academia,  industry, 
project  managers,  military  service  and 
OSD  staff  offices  to  ascertain  what 
methodologies  are  being  utilized  for 
managing  risk  and  uncertainty. 

21.  Determine  the  most  appropriate  metho¬ 
dologies  for  the  front-end  analysis  in  a 
program  and  identify  areas  for  further 
research. 

22.  Identify  methodologies  to  analyze  risk 
and  uncertainty  in  human  factors  and 
recommend  areas  for  further  research. 

23.  Special  emphasis  needs  to  be  placed  on 
determining  risk  and  uncertainty  related 
to  the  variable  of  "performance"  since 
it  is  a  much  more  complex  methodology. 
Initiate  research  on  selected  trial  ap¬ 
plications  for  projects  having  specific 
requirements . 

24.  Develop  methods  to  transfer  the  state-of- 
the-art  in  risk  and  uncertainty  analysis 
management  to  the  system  acquisition 
practitioners.  This  education  would 
include  the  ideas,  structures  and  tools 
(application,  definitions,  interrelation¬ 
ships,  methodologies  and  assessments). 

25.  Network  models  like  the  VERT  and  RISCA 
developed  for  the  U.S.  Army  can  assist  a 
manager  in  dealing  with  the  risks  asso¬ 
ciated  with  complex  acquisition  programs 
in  other  DoD  areas. 

26.  DoD  program  sponsors,  project  managers. 


222  .  3 


directors  (RDT&E)  ,  should  have  cos.t  anal¬ 
ysts  to  ensure  that  the  requirements  for 
the  identification  areas  of  risk  and  un¬ 
certainty  under  USDRE  5000.2  and  the 
DCP  s  for  milestone  reviews  are  ade¬ 
quately  treated  in  planning  and  properly 
reviewed . 
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ABSTRACT 


In  today's  competitive  environment,  industry 
and  the  Government  have  become  increas¬ 
ingly  aware  of  the  need  to  assess  risk  as 
part  of  the  decision-making  process.  The 
probability  of  achieving  the  estimate,  range 
of  probable  deviation,  and  enumeration  of 
possible  causes  for  overrun  must  be  evalu¬ 
ated  before  a  knowledgeable  decision  can  be 
made  (1,  3,  5,  8). 

This  paper  presents  the  cost-schedule-tech¬ 
nical  risk  assessment  technology  being  used 
by  Boeing  Vertol  for  the  past  eight  years  to 
evaluate  proposed  program  estimates  and  the 
estimates  to  complete  of  on-going  programs. 
The  complete  risk  assessment  process  is  des¬ 
cribed  and  several  useful  applications  are 
discussed . 


INTRODUCTION 


Boeing  Vertol  is  a  major  manufacturer  of  heli¬ 
copters.  We  are  a  high  technology  company, 
as  are  most  companies  within  the  aerospace 
industry.  The  risk  assessment  methodology 
presented  herein  evolved  from  a  need  to 
understand  the  risk  associated  with  major 
technical  development  programs.  It  has 
since  proved  applicable  to  production  pro¬ 
grams,  smaller  research  and  development 
contracts,  special  analyses  of  technical  trade¬ 
offs,  proposed  product  warranties,  and 
incentive  fee  arrangements . 

The  methodology  has  been  successfully 
applied  to  major  aircraft  programs  of  other 
Boeing  Companies.  At  the  Boeing  Military 
Aircraft  Company  in  Wichita,  Kansas,  major 
B-52  program  cost  proposals  were  evaluated, 
including  the  Weapons  System  Trainer  devel¬ 
opment  and  later  the  proposed  production 
program;  the  Offensive  Avionics  System 
(OAS)  and  the  Cruise  Missile  Integration 
(CMI)  Full  Scale  Development  programs;  and 
the  initial  production  programs  for  OAS/CMI. 
The  C-14  Program  cost  estimate  was 
thoroughly  evaluated  for  the  Boeing  Aero¬ 
space  Company  in  Seattle,  Washington. 

It  should  be  noted  that  many  different  per¬ 
sonnel  at  both  Vertol  and  other  Boeing  loca¬ 
tions  have  been  able  to  provide  meaningful 
risk  data  for  these  assessments.  Thereby 


i  providing  some  measure  of  proof  regarding 
the  universal  application  of  the  Boeing  Vertol 
risk  assessment  technique.  The  methodology 
combines  a  realistic  approach  for  obtaining 
relatively  unbiased  personal  judgement  data 
with  an  innovative  computer  model  that  simu¬ 
lates  the  proposed  program  expenditures. 
This  methodology  is  unique  because  it  does 
not  utilize  a  decision  tree  or  network 
approach,  which  in  other  methods  becomes 
complicated  and  unrealistic  when  applied  in 
sufficient  detail  to  obtain  a  comprehensive 
analysis. 

A  basic  concept  of  the  methodology  is  that 
inherent  program  risk  is  primarily  attribut¬ 
able  to  the  occurrence  of  unforeseen  prob¬ 
lems  which  are  usually  technical  in  nature 
(1,  8,  'll).  Cost  and  probabi  I  istic  data, 

regarding  potential  problems,  are  obtained 
from  persons  with  the  appropriate  expertise. 
The  computer  model  simulates  the  occurrence 
of  these  problems  to  derive  probability  dis¬ 
tributions.  These  distributions ,  when  com¬ 
bined  by  simulation  with  other  subjectively 
defined  probability  distributions,  provide  a 
quantification  of  program  risk.  The  results 
of  the  analysis  include  risk  information 
regarding  potential  problems  and  cumulative 
probability  curves  that  provide  valuable  data 
for  company  and  customer  management  deci¬ 
sion-making.  In  addition,  it  serves  as  the 
basis  for  risk  reduction  activities  during  the 
period  of  contract  performance. 


METHODOLOGY 


Risk  assessment  is  usually  performed  after 
the  proposed  program  has  been  defined. 
Work  statements,  work  breakdown  structure, 
schedules  or  networks,  and  an  initial  estimate 
are  generally  available  at  this  stage  of  pro¬ 
posal  preparation.  This  program  planning 
and  estimating  data  appears  at  the  top  of  the 
Risk  Assessment  Flow  Diagram  (Figure  1)  for 
a  typical  prototype  aircraft  program.  The 
diagram  illustrates  the  complete  assessment 
process  including  both  the  gathering  of  risk 
data  and  utilization  of  the  computer  simula¬ 
tion  model. 

Initially,  the  proposed  program  is  divided 
into  major  phases.  These  are  again  subdi¬ 
vided  into  no  more  than  20  to  30  program 
elements  to  accomplish  the  analysis  in  suffi¬ 
cient  detail  without  being  "bogged  down"  by 
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RISK  ASSESSMENT  FLOW  DIAGRAM 

FIGURE  1 


over  complexity.  This  permits  determination 
of  risk  information  with  the  personnel  most 
knowledgeable  of  the  particular  elements  of 
the  program.  Figure  2  provides  an  example 
of  the  major  phases  and  elements  for  the 
prototype  aircraft  program. 

Two  alternate  approaches  are  utilized  to 
assess  the  risk  in  each  program  element. 
The  first,  shown  on  the  left  side  on  Figure 
1 ,  employs  simulation  to  evaluate  the  impact 
of  potential  technical  problems.  This  tech¬ 
nique  is  used  for  analysis  of  program  ele¬ 
ments  considered  to  have  high  risk.  The 
other  approach  (on  the  right  side  of  Figure 
1)  consists  of  a  subjective  assessment  of 
probability  distributions  (9).  It  is  used  for 


elements  with  lower  risk  or  when  the  model¬ 
ing  of  potential  problems  is  not  feasible. 
The  probability  distributions  for  each  pro¬ 
gram  element,  developed  by  either  approach, 
are  merged  or  summed  by  Monte  Carlo  simu¬ 
lation  (6,  7,  10).  The  two  approaches  and 
the  summation  process  are  more  fully  des¬ 
cribed  in  the  following  paragraphs. 


SIMULATION  OF  POTENTIAL  PROBLEMS 


This  technique  is  used  to  assess  the  higher 
risk  elements  of  the  program  (as  noted  by 
asterisks  in  Figure  2).  Generally,  the  risk 
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PROTOTYPE  AIRCRAFT  PROGRAM 

,  COST  ESTIMATE 

(dollars  in  millions) 

PHASE  I  -  AIR  VEHICLE  DESIGN 

$4.0 

1.  Airframe  &  Landing  Gear 

2.  Propulsion  &  Drive  System 

3.  Vehicle  Subsystems 

4.  SAS,  Flight  Control  System 

5.  Other  Design  &  Support 
*6.  Design  Support  Tests 

PHASE  II  -  MAJOR  SUBCONTRACTOR 

$5.0 

7.  Wing  and  Nacelle 

PHASE  III  -  A/C  MANUFACTURING 

$9.0 

4» 

8.  Manufacturing/Final  Assembly 

9 ,  Tool i ng 

10.  Material 

1 1 .  Engineering  Liaison 

PHASE  IV  -  SUBSYSTEM  TEST 

$3.0 

*1 2 .  Component  Test 

*13.  Propulsion  System  Test 

PHASE  V  -  GROUND  &  WIND  TUNNEL  TEST 

$2.0 

*14.  Ground  Tie-Down  Test 
*15.  Wind  Tunnel  Test 

PHASE  VI  -  AIRCRAFT  FLIGHT  TEST 

$3.0 

1 6.  Preparation/Instrumentation 
*17.  Boeing  FI ight  Test 

18.  Customer  Flight  Test 

PHASE  VII  -  PROJECT  MGT .  &  OTHER 

$2.0 

19. 

Proj  ect 

Management 

20. 

Data  & 

Documentation 

21  . 

Spares 

&  Miscellaneous 

Total  Program  Cost  $28.0 
^Simulate  Impact  of  Potential  Problems 

MAJOR  PHASES  AND  ELEMENTS 
FIGURE  2 


1 


or  technical  problems  surface  in  the  testing 
elements  such  as  Tool  Proving,  Bench  Test 
or  Operational  Test,  although  the  cause  of 
the  problems  may  be  attributable  to  a  prior 
design,  analytical,  or  fabrication  element. 

The  risk  analyst  asks  the  engineers  and  line 
supervisors  to  first  identify  potential  techni¬ 
cal  problems  that  could  occur  in  these  higher 


risk  elements  as  shown  in  Figure  3,  Techni¬ 
cal  Risk  Identification.  Other  data  provided 
includes  a  judgmental  evaluation  of  the  pro¬ 
blem  probability  of  occurrence  and  its  aver¬ 
age  cost/scheduie  impact.  The  problem  list¬ 
ing  is  usually  organized  by  subsystem  and 
indicates  the  program  elements  where  each 
problem  could  occur  (Source  of  Discovery). 
This  list  gives  to  program  management,  a 
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PRJTUTfPE  AIRCRAFT  PflOGRM 

Oft  I VE  SYSTEM 

POTENTIAL  PROBLEM 

SOURCE  OF 
OISCOVERY 

PPOB'LY 
OF  OCCUR 

CONSEQUENCE 

IMPACT 

COST  i 
SCHEDULE 

RECOVERY  ACTION 

TRANSMISSION 

COOLING  IS 

INADEQUATE 

PROPULSION 

TEST 

21 

TRANSMISSION  OYERHEAT 
LIMITS  OPERATION 

swtc 

2  WHS 

CHANGE  TO  OIL  PUMP 

WITH  INCREASED 

CAPACITY 

GEAR  C0N0ITI0N 
UNSATISFACTORY  AT 

TEAR DOWN  INSPECTION 

PROPULSION 

TEST 

51 

SEART00TH  SURFACE 

DAMAGE  (SCUFFING) 

$50* 

2  WHS 

REVISE  LUBE  SYSTEM, 
REGRINO  GEARS,  AND 

RETEST 

GR0UN0 

TEST 

SI 

$7SK 

B  WHS 

CROSS-SHAFT 

SUPPORT  FAILURE 

PROPULSION 

TEST 

21 

LOSS  OF  CROSS-SHAFT 
CONTINUITY  LIFT 
UNBALANCE  t  CONTROL. 

$acK 

3  WHS 

DESIGN  MOO  FOR 

INCREASED  STRENGTH. 
REPAIR  i  REPLACE 
COMPONENTS  IF  REQ'O. 

GROUND 

TEST 

11 

$200H 

B  WHS 

LOW  SEARING 

LIFE 

PROPULSION 

TEST 

SI 

LOW  TRANSMISSION 
OVERHAUL  TOUR 

$20H 

PURCHASE  AOO’L  SPARES. 
REVISE  INSTALLATION 
PROCEDURES,  LOCK  BRG 
RACE. 

CLUTCH  FAILURE 

PROPULSION 

TEST 

SI 

LOSS  OF  DRIVE 

FROM  ONE  ENGINE 

$28X 

1  WK 

LUBE  SYSTEM  REVISIONS 

GR0UN0 

TEST 

2% 

$7SH 

3  WHS 

TRANSMISSION 

MOUNTING  LUGS 

FAILURE 

GROUND 

TEST 

11 

INADEQUATE 

FATIGUE  LIFE 

$40CK 

9  WHS 

REDESIGN  FOR  ADO'L. 
STRENGTH.  LAB  RETEST 
&  GROUND  TEST 

PARTIAL  REPEAT. 

TECHNICAL  RISK  IDENTIFICATION 
FIGURE  3 


preview  of  the  risk  envisioned  by  line  super¬ 
vision.  A  thorough  review  at  this  point  in 
time  can  result  in  redefining  some  aspects  of 
the  program  to  reduce  the  risk. 

Next,  the  analyst  conducts  a  potential  pro¬ 
blem  modeling  session  with  the  person  who 
identified  the  problem,  to  obtain  more  data. 
A  small  (optimistic)  and  large  (pessimistic) 
cost/schedule  impact  are  determined.  This 
requires  a  brief  one  sentence  description  and 
basic  estimate  information  for  each  impact  so 
that  a  valid  cost  estimate,  including  engi¬ 
neering,  manufacturing,  retest,  etc.  can  be 
prepared.  Also,  a  relative  probability  is 
assigned  to  each  of  the  three  problem 
impacts,  including  the  previously  defined 
average  impact.  This  data  gathering  can  be 
accomplished  rapidly  once  the  person  has 
experienced  an  initial  modeling  session.  The 
work  sheet  resulting  from  a  typical  problem 
modeling  session  is  provided  in  Figure  4. 

After  the  problems  have  been  modeled,  they 
are  regrouped  by  source  of  occurrence  and 
reviewed  with  the  manager  of  the  particular 
element  involved.  For  example,  the  Wind 
Tunnel  manager  would  review  the  potential 
problems  that  could  occur  during  Wind  Tun¬ 
nel  Test.  He  ranks  the  problems  relative  to 
his  opinion  of  their  probability  of  occurrence 
by  sorting  the  modeling  data  work  sheets. 


He  then  reviews  and  revises  the  previously 
identified  probabilities  of  occurrence,  as 
required.  This  step  is  a  further  check  to 
minimize  any  bias  of  the  individuals  who  ini¬ 
tially  assigned  a  probability  of  occurrence  to 
each  problem  during  the  identification  pro¬ 
cess.  Next,  he  provides  additional  risk  data 
by  determining  the  probability  that  if  a  pro¬ 
blem  occurs,  it  was  previously  identified. 
This  probability  generally  ranges  from  60%  to 
85%.  An  unidentified  problem  with  associated 
cost/schedule  impacts  and  probabilities  is 
also  modeled  with  the  help  of  the  risk  analyst 

The  last  step,  in  the  modeling  process,  is  an 
evaluation  of  one  of  the  more  important  para¬ 
meters.  The  supervisor  is  asked  to  estimate 
the  most  likely  number  of  problems  expected 
to  occur  during  the  schedule  span  of  the 
program  element.  In  making  this  estimate, 
he  considers  the  problems  encountered  on 
past  programs  and  problems  modeled  for  this 
particular  program  and  their  probability  of 
occurrence.  This  risk  parameter  is  relative¬ 
ly  easy  to  obtain  since  the  supervisor  can 
relate  to  "the  number  of  problems  that  may 
occur"  and  therefore,  he  can  feel  comfortable 
in  making  such  a  decision.  Also,  the  para¬ 
meter  can  be  readily  reviewed  by  program 
management,  later,  when  the  analysis  results 
are  presented.  An  example  of  the  Potential 
Problem  Modeling  Data  for  a  program  element 
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PROTOTYPE  AIRCRAFT  PROGRAM 

PROBLEM  NO.  10  ESTIMATE  WORK  SHEET  PROB'LY  OF  OCCUR  2% 


DESCRIPTION 

Drive  System 

OPTIMISTIC 

Inadequate  airflow  at 
maximum  fan  RPM. 

Change  to  fan  with 
greater  CFM  capacity. 

AVERAGE 

Oil  delivery  rate  to 
cooler  precludes 
optimum  cooler  perfor¬ 
mance.  Change  to  pump 
w/increased  capacity. 

2  week  retest  req  1  d . 

PESSIMISTIC 

Total  cooler  capacity 
is  inadequate.  Req. 
new  cooler  and  assoc, 
equi pment . 

4  week  retest  req  1  d . 

Transmission  cooling  is 
inadequate. 

PROPULSION  SYSTEM  TEST 

{Program  continues  wit 

l  operating  restrictions) 

Schedule  impact  to  resolve. 

Relative  probability. 

1  Week 

20X 

2  Weeks 

m 

5  Weeks 

40% 

COST  IMPACTS 

650  HRS  $13,000 

1000  HRS  $20,000 

1300  HRS  $26,000 

Eng i nee  ri ng 

Manuf actu  ri ng 

[  550  HRS  $1 1 ,000 

900  HRS  $18,000 

1400  HRS  T28.000 

Material 

$  5,000 

$10,000 

SToTooo 

Total  Oollars 

$29,000 

$43,000 

$94,000 

PROBLEM  MODELING  WORK  SHEET 
FIGURE  4 


is  provided  in  Figure  5.  Note  that  a  "No 
Problem"  estimate  prepared  by  the  risk  anal¬ 
yst  is  shown  on  the  third  line,  right  side  of 
the  form.  To  calculate  this  estimate,  the 
program  element  estimate  is  reduced,  to 
"scrub  out"  the  cost  attributable  to  problems 
inherent  in  the  statistical  data  which  served 
as  the  basis  for  its  determination.  This  is 
necessary,  otherwise  the  simulation  model 
would  add  the  cost  increase  due  to  problem 
occurrence  on  top  of  an  estimate  which 
already  contains  the  cost  of  significant  pro¬ 
blems  . 

The  Simulation  of  Potential  Problems  Flow 
Diagram  presented  in  Figure  6  depicts  the 
simulation  process  for  a  program  element. 
The  simulation  method  employed  uses  a  com¬ 
puter  generated  random  probability,  to  pick 
a  corresponding  value  or  action  from  discrete 
probability  distributions  formed  from  the 
problem  modeling  data.  First,  a  Poisson 
probability  distribution,  based  on  the  most 
likely  number  of  problems  expected  to  occur 
over  the  element  schedule  span,  is  tested  to 
see  if  a  problem  has  occurred.  If  not,  the 
computer  proceeds  to  the  next  month  and 
tests  again.  Assuming  a  problem  is  found  to 
have  occurred,  a  choice  is  made  as  to 
whether  it  is  an  identified  or  unidentified 
problem.  If  it  is  "identified",  one  of  the 
particular  identified  problems  is  selected  and 
also  one  of  three  possible  impacts.  If  an  un¬ 
identified  problem  is  selected,  one  of  five 
possible  impacts  is  chosen.  The  problem  cost 
impact  is  added  to  the  "No  Problem"  Baseline 
Estimate.  The  schedule  remaining  is  tested 
for  sufficient  time  to  resolve  the  problem  and 


extended  as  necessary.  Finally,  the  proba¬ 
bility  of  the  selected  problem  impact  occur¬ 
ring  again  is  reduced  and  the  computer  pro¬ 
ceeds  to  the  next  month.  The  model  contin¬ 
ues  to  simulate  the  "real  life"  of  a  program 
element  starting  with  the  first  month  and 
proceeding  month  by  month  to  the  end  of  the 
schedule  span.  The  result  is  a  sample  case 
of  cost  and  schedule  information.  The  prob¬ 
ability  distributions,  required  later  in  the 
model,  are  computed  by  simulating  several 
hundred  sample  cases  for  each  program  ele¬ 
ment  (6,  10). 

Of  special  interest  is  the  initial  problem 
occurrence  determination  which  is  accom¬ 
plished  by  sampling  a  Poisson  probability 
distribution  (2,  7)  to  make  this  monthly 
selection.  The  "most  likely  number  of  pro¬ 
blems  expected  to  occur"  during  the  sche¬ 
dule  span  is  used  as  the  basic  parameter  to 
calculate  the  probability  of  0,  1,  2  or  3  pro¬ 
blems  occurring  in  any  month.  Generally, 
the  probability  of  zero  or  no  problems  occur¬ 
ring  is  quite  high  and  one  problem  occurring 
relatively  small.  Two  and  three  problems 
occurring  generally  have  very  small  proba¬ 
bilities.  Note  that  the  probability  of  "3  pro¬ 
blems  occurring"  include  the  residual  proba¬ 
bility  of  4,  5,  ...  problems  occurring.  If 

the  element  schedule  span  lengthens  due  to 
simulated  problem  occurrence,  the  model 

"winds  down"  because  the  Poisson  distribu¬ 
tion  is  recalculated  based  on  the  longer 
span.  The  probability  of  no  problems  occur¬ 
ring  becomes  even  larger  and  the  probability 
of  1 ,  2  or  3  problems  occurring  becomes  cor- 
I  respondingly  smaller.  These  features  enable 
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Progr— :  PROTOTYPE  AIRCRAFT  PROGRAM 


Pt»s«:  PROPULSION  SYSTEM  TEST 


Expect*  Problem  „  Schrtult  In  Weeks:  12  Prob'ly  Identified  Problem:  ,60 


(Identified:  _2._4  I  Unidentified:  1 .6  )  "No  Problem"  Estimate:  $11 00.  CO 


No 

Proble*  Description 

Prob’ly 

Potential  Problem  Impact 

of  Occur 

Oescr. 

Prob'ly 

Estimate* 

] 

Actuator  -  False 

.20 

Opt. 

.30 

50.0 

1 

Failure  Signals 

Avg. 

Pess. 

.50 

.20 

70.0 

100.0 

2 

3 

2 

Engine  Overtemp 

.10 

Opt. 

.20 

20.0 

1 

(External ) 

Avg. 

.60 

20.0 

1 

Pess. 

.20 

40.0 

2 

3 

Engine  Fire  Extinguisher 

.10 

Opt. 

.30 

20.0 

1 

Inadequate 

Avg. 

.50 

25.0 

1 

Pess. 

.20 

40.0 

2 

Nacelle  Hinge  - 

.05 

Opt. 

.40 

43.0 

2 

4 

Excessive  Wear 

Avg. 

.50 

63.0 

3 

Pess. 

.10 

88.0 

4 

5 

Engine  Inlet  Flow 

Olstortlon 

.05 

Opt. 

Avg. 

.30 

.50 

25.0 

38.0 

2 

3 

Pess. 

.20 

63.0 

5 

Gear  Condition 

Opt. 

.60 

36.0 

1 

6 

Unsatisfactory  at 

.05 

Avg. 

.30 

50.0 

2 

Teardovm  Inspection 

Pess. 

.10 

80.0 

3 

7 

Low  Searing  Life 
(Orlve  System) 

.05 

Opt. 

Avg. 

Pess. 

.50 

.40 

.10 

10.0 

20.0 

78.0 

0 

0 

3 

Opt. 

.30 

23.0 

1 

8 

Clutch  Failure 

.05 

Avg. 

.60 

28.0 

1 

Pess. 

.10 

45.0 

1 

Control  System 

.02 

Opt. 

.20 

28.0 

1 

9 

Actuator  Chatter 

Avg. 

.60 

33.0 

1 

Pess. 

.20 

58.0 

1 

10 

Transmission  Cooling 

Is  Inadequate 

.02 

Opt. 

Avg. 

Pess. 

.20 

.40 

.40 

23.0 

48.0 

94.0 

1 

2 

5 

Optimistic 

.05 

30.0 

1 

Less  Optimistic 

.20 

60.0 

2 

Unidentified  Problem 

Average 

.60 

100.0 

3 

Less  Pess. 

.10 

250.0 

4 

♦Dollars  In  Thousands 

Pessimistic 

.05 

600.0 

6 

POTENTIAL  PROBLEM  MODELING  DATA 


FIGURE  5 


the  potential  problem  simulation  to  essentially 
duplicate  the  occurrence  of  actual  problems 
similar  to  experience  on  past  programs. 


SUBJECTIVE  PROBABILITY  EVALUATION 


This  approach  consists  of  a  subjective 
assessment  of  probability  distributions  (9) 
using  Normal  probability  graph  paper  as 
"talking  paper"  during  the  information  gath¬ 
ering  process.  It  is  applied  to  program  ele¬ 
ments  considered  to  have  moderate  risk  or 
where  potential  problem  modeling  is  not  fea¬ 
sible,  such  as  Program  Management.  Opti- 

.3 

* 
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mistic,  most  likely,  and  pessimistic  estimates 
of  program  elements  and  associated  probabil¬ 
ities  are  used  to  construct  a  probability 
curve  connecting  three  or  more  points. 
These  are  reviewed  for  reasonableness  by 
checking  such  factors  as  the  variation  of 
slope,  probability  of  achieving  the  estimate, 
and  relationship  to  past  experience.  Fig¬ 
ure  7  is  a  probability  curve  derived  by  this 
subjective  process.  When  applicable,  an 
adjustment  should  be  made  to  the  curve  to 
"scrub  out"  the  historical  cost  impact  of 
problems  simulated  in  other  program  ele¬ 
ments.  For  example,  Manufacturing  and 
Final  Assembly  (Element  No.  8)  should  be 
reduced  to  delete  the  cost  attributable  to 
refabrication  of  test  parts  on  past  programs. 
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FIGURE  6 


SUBJECTIVE  PROBABILITY  CURVE 


FIGURE  7 
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SUMMATION  OF 

PROBABILITY  DISTR  I  BUT  IONS 


Probability  distributions  for  each  program 
element  are  derived  by  either  simulating  the 
impact  of  potential  problems  or  subjectively 
evaluating  probability  curves.  These  distri¬ 
butions  are  summed  by  a  simulation  tech¬ 
nique  that  randomly  selects  a  cost  estimate 
from  each  element  probability  curve  to  form 
a  typical  program.  Several  hundred  sample 
cases  are  computed  to  determine  a  statisti¬ 
cally  valid  probability  distribution  for  each 
phase  and  the  total  program  (6,  10).  A 

unique  feature  of  this  summation  process  is 
the  application  of  a  computer  generated  cor¬ 
rection  factor  to  negate  the  statistical  con¬ 
vergence  which  occurs  in  accordance  with 
Sampling  Theory  and  the  Central  Limit 
Theorm  (2,  12).  The  personnel  using  their 
judgement  to  "model"  problems  and  to  deter¬ 
mine  subjective  probability  curves  do  not 
think  in  terms  of  their  estimate  being  a  small 
sample,  with  the  potential  for  large  cost 
deviations.  This  is  easily  demonstrated  by 
dividing  a  program  into  twice  the  number  of 
elements  and  modeling  problems  and  subjec¬ 
tive  curves.  The  summation  results  will 
show  a  considerable  reduction  in  the  stand¬ 
ard  deviation  which  is  not  valid.  The  cor¬ 
rection  factor  restores  this  loss  in  risk 
deviation  while  still  retaining  the  arithmetic 
average  of  the  distribution . 


evaluation  of  risk  curves 


Fundamental  to  the  utilization  of  risk  assess¬ 
ment  in  the  decision  process  is  the  evalua¬ 
tion  of  the  resultant  cumulative  probability 
distribution  curves.  An  example  of  these 
curves  for  the  Prototype  Aircraft  Program  is 
shown  in  Figure  8.  The  three  probability 
curves  on  the  graph  show  the  cumulative 
impact  of  potential  problems  as  follows: 

1)  The  "no  problem"  baseline  estimate  pro¬ 
vides  a  reference  for  evaluating  the  cost 
impact  of  potential  problems. 

2)  The  identified  risk  area  defines  the 
range  of  anticipated  cost  attributable  to 
the  occurrence  of  the  potential  problems 
identified  in  the  information  gathering 
process.  It  also  includes  the  variation 
above  50%  probability  of  the  subjective 
distributions  relative  to  a  "straight  line" 
cumulative  Normal  probability  distribu¬ 
tion  . 

3)  The  additional  impact  of  the  unidentified 
potential  problems  is  shown  by  the 
upper  line.  This  "total  risk"  probabil¬ 
ity  curve  is  more  subjective  than  the 
identified  problem  curve,  due  to  the 
lack  of  precise  definition.  However,  it 
provides  an  upward  boundary  reference 
of  potential  program  cost  if  unforeseen 
problems  occur. 
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FIGURE  8 
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Interpretation  of  probability  curves  is  a  sub¬ 
jective  evaluation  based  on  personal  experi¬ 
ence.  Where  one  manager  considers  a  50% 
probability  of  achieving  the  estimate  as 
acceptable,  another  manager  may  look  at  a 
75%  probability  as  the  lowest  to  which  he  can 
agree.  The  risk  analyst  must  deal  with 
these  personal  opinions  on  an  individual 
basis.  Any  interpretation  should  recognize 
that  assuming  the  program  could  actually 
happen  several  hundred  times,  the  curves 
provide  a  picture  of  the  final  cost  versus 
cumulative  probability  of  occurrence.  The 
point  where  the  horizontal  cost  estimate  line 
intersects  the  curve  provides  the  probability 
of  not  overrunning  the  estimate.  Therefore, 
this  probability  should  be  viewed  in  relation 
to  the  slope  of  the  curve.  For  example,  if  a 
high  probability,  such  as  the  90%  confidence 
level,  indicates  a  relatively  small  increase  in 
cost,  then  the  probability  of  achieving  the 
estimate  has  less  significance.  The  interpre¬ 
tation  being  that  even  if  an  overrun 
occurred,  it  would  be  very  small. 

Most  cost  estimating  is  based  on  statistics 
derived  from  past  programs.  The  estimates 
prepared  from  this  statistical  data  base 
include  an  average  cost  impact  of  problems 
that  occurred  on  those  programs.  The  risk 
analysis,  on  the  other  hand,  requires  a 
hypothetical  "no  problem1*  baseline  which  was 
obtained  by  subjectively  “scrubbing  out"  an 
average  cost  for  background  problems.  The 
risk  model  then  simulates  the  cost  impact  of 
specifically  identified  problems  for  a  particu¬ 
lar  program  to  determine  a  probability  distri¬ 
bution  that  portrays  the  cost  risk  associated 
with  its  statement  of  work.  Thus,  if  the 
program  being  estimated  is  typical  of  the 
past  programs,  with  similar  expected  pro¬ 
blems,  there  should  be  a  50%  probability  of 
achieving  the  estimate.  However,  for  a  pro¬ 
gram  which  is  not  typical  of  the  past  his¬ 
tory,  the  probability  of  achieving  the  esti¬ 
mated  cost  may  vary  significantly  from  the 
50%  point. 


the  risk  and  where  risk  reduction  activ¬ 
ities  could  be  effectively  applied. 

o  Cumulative  probability  curves  (Figure  8) 
which  display  the  probability  of  achiev¬ 
ing  the  estimated  cost  and  the  range  of 
probable  deviation.  Enables  manage¬ 
ment  to  understand  the  extent  of  pos¬ 
sible  overruns  for  the  total  program 
and  specific  phases. 

This  review  can  result  in  follow-up  actions. 
For  example,  if  a  later  phase  such  as  Flight 
Test  indicates  an  excessively  high  risk,  then 
consideration  is  given  to  performing  addi_ 
tional  laboratory  and  wind  tunnel  tests  ear¬ 
lier  in  the  program  to  surface  problems  in 
sufficient  time  to  resolve  them  at  a  lower 
cost  and  with  minimum  schedule  delay. 
Another  example  is  high  risk  in  a  phase  pri¬ 
marily  derived  from  subjectively  defined 
probability  curves,  such  as  Manufacturing . 
This  could  be  an  indication  that  the  base 
estimate  is  too  low  and  should  be  revised. 

The  total  program  probability  curve  can  be 
used  in  determining  the  final  cost  estimate 
for  a  Government  contract.  This  decision 
must  be  made  within  the  context  of  the  com¬ 
petitive  situation,  type  of  contract  (e.g., 
cost  vs.  fixed  price),  and  compliance  with 
Government  regulations.  If  the  probability 
of  achieving  the  estimate  was  only  40%,  the 
90%  confidence  estimate  was  not  excessively 
high,  and  the  proposed  contract  will  be  an 
incentive  fee  cost  type,  then  the  estimate 
could  be  considered  as  an  acceptable  risk  in 
a  highly  competitive  environment.  On  the 

other  hand,  for  fixed  price  quotations  and 
commercial  proposals,  the  cost  estimate  is 
normally  expected  to  have  a  higher  than  50% 
confidence  of  being  achieved.  Thus,  risk 

assessment  makes  a  significant  contribution 

to  pricing  strategy  and  the  ultimate  bid 

price  decision. 
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MANAGEMENT  REVIEW  AND  ACTION 


Management  review  of  a  risk  assessment 
includes  an  evaluation  of  the  essential  data 
used  in  the  analysis  and  probability  curves 
for  the  total  program  and  major  phases. 
The  data  evaluated  consists  of; 

o  Definition  of  program  phases,  elements 
and  associated  cost  estimates  (Figure  2). 
Presents  how  the  analysis  was  organized 
and  provides  the  base  estimates. 

o  Listing  of  potential  problems  identified 
and  modeled  for  the  analysis  (Figure  3). 
Provides  visibility  as  to  what  is  causing 


INCENTIVE  FEE  STRATEGY 


Risk  analysis  probability  curves  can  provide 
substantiation  for  proposing  incentive  fee 
arrangements  (13).  Figure  9  is  an  example 
of  how  the  Prototype  Aircraft  Program  pro¬ 
bability  curve  (Figure  8)  can  be  used  to 
establish  the  target  cost,  and  the  cost 
values  at  minimum/maximum  fee  limits. 
Assume  it  was  decicjed  to  propose  a  cost  of 
$30  million  and  a  fee  of  11%  ($3.3  million)  for 
the  target  cost.  The  10%/90%  cumulative 
probability  values  of  the  Total  Risk  Curve  in 
Figure  8  which  are  $26  million  and  $37  million 
respectively,  are  used  to  define  the  fee  shar¬ 
ing  limits.  The  maximum  fee  of  15%  and 
minimum  fee  of  4%  relative  to  these  corres¬ 
ponding  cost  values  determine  an  incentive 
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PROTOTYPE  AIRCRAFT  PROGRAM 

(DOLLARS  IN  MILLIONS) 


$26.0  COST  AT 
MAXIMUM  FEE 


TYPICAL  INCENTIVE  FEE  PROPOSAL 
FIGURE  9 


share  line  of  70%/30%  (customer/contractor ) . 
For  example,  if  the  actual  contract  cost 
turns  out  to  be  $29  million,  then  the  cus¬ 
tomer  saved  $1  million  in  cost  and  pays  $.3 
million  of  this  saving  to  the  contractor  as 
incentive  fee  for  underrunning  the  contract 
cost.  The  important  feature  of  relating  the 
incentive  fee  share  line  to  the  risk  proba¬ 
bility  distribution  is  that  the  probability  of 
earning  various  fee  (profit)  dollars  can  be 
readily  determined.  Thus,  the  program  risk 
assessment  curve  provides  assistance  in 
establishing  the  strategy  for  the  proposed 
incentive  fee  arrangement  and  subsequent 
negotiations . 


SPECIAL  USE  ANALYSES 


described,  yields  the  required  cost  proba¬ 
bility  distributions.  This  risk  data  aids  in 
developing  the  final  company  position. 

Another  use  of  risk  analysis  is  to  augment 
the  decision  data  for  technical  trade-off 
analysis  of  alternative  configurations  and 
designs.  This  is  usually  limited  to  major 
"trade-offs",  such  as  choosing  between  alter¬ 
native  rotor  blade  designs.  In  these  evalua¬ 
tions,  technical  considerations  are  very 
important.  However,  risk  analysis  gives 
insight  into  the  technical  problems  that  could 
occur  and  provides  another  dimension  to 
evaluating  the  cost  estimates. 


RISK  REDUCTION 


•+ 


h 


Risk  assessment  is  useful  in  establishigg 
product  warranties  for  high  value  and/or 
safety  of  flight  components.  These  warran¬ 
ties  usually  relate  to  the  reliability  and  main¬ 
tainability  of  dynamic  components  (e.g., 
rotor  blades,  transmissions,  actuators, 
etc.).  Since  the  occurrence  of  technical 
problems  is  the  cause  of  failing  to  meet  a 
warrantee,  the  simulation  of  these  potential 
problems  using  the  methodology  previously 


Risk  reduction  is  a  continuing  activity.  The 
problems  identified  during  the  preparation  of 
a  proposal  risk  analysis  enables  management 
to  revise  program  requirements  and  schedules 
to  reduce  potential  risk.  Fallback  positions 
are  formulated  to  minimize  the  impact  of  the 
problems.  The  probability  data  and  curves 
are  used  to  compare  the  cost  of  added  test¬ 
ing  to  the  resultant  reduction  in  risk. 
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risk  avoidance  action 
figure  TO  . 


Once  the  contract  has  been  received,  the 
potential  problems  can  be  monitored  periodic¬ 
ally.  Figure  10  presents  a  typical  format 
used  on  a  previous  program  for  this  purpose. 
Note  that  as  the  risk  avoidance  actions  were 
recorded  chronologically,  the  probability  of 
occurrence  was  subsequently  reduced.  As 
the  program  progresses,  cost  estimates  for 
the  remaining  work  are  prepared.  The  risk 
analysis  is  updated  when  these  estimates 
indicate  a  significant  increase  in  cost  or 
after  a  phase  of  the  program  has  been  com¬ 
pleted  (e.g.,  Design  Completion).  Problems 
that  occurred  and  were  resolved  are  elimi¬ 
nated.  Problems  that  did  not  occur  in 
already  completed  program  phases  are  also 
deleted.  The  probability  of  problem  occur¬ 
rence  might  be  revised.  New  problems  may 
be  identified.  These  “estimate  to  complete” 
risk  analyses  provide  much  greater  visibility 
into  program  status  than  “single  point”  esti¬ 
mates  . 

The  most  important  aspect  of  risk  analysis 
and  avoidance  activities  is  the  attitude  of 
risk  awareness  that  is  created.  Although 
too  intangible  to  evaluate  quantitatively ,  risk 
awareness  soon  permeates  throughout  the 
organization.  The  result  is  threefold;  it 
reduces  the  time  to  conduct  an  analysis,  it 
improves  accuracy  and  more  importantly,  it 
has  a  definite  benefit  in  actions  to  reduce 
risk . 


RISK  ASSESSMENT  VALIDATION 


Validation  of  risk  assessments  can  be  deter¬ 
mined  by  comparative  evaluation,  although 
precise  measurement  is  difficult  since  a  sin¬ 
gle  program  "at  completion"  cost  is  being 
compared  to  a  probability  distribution  with  a 
continuum  of  cost  values.  Certainly  if  the 
"at  completion"  cost  exceeds  the  100%  proba¬ 
bility  value,  which  is  hardly  ever  the  case, 
it  was  a  poor  risk  assessment.  In  addition, 
pricing  strategy,  contract  terms,  success  (or 
failure)  in  negotiating  cost  targets  for  the 
initial  contract  and/or  contract  changes,  and 
excessive  inflation  impact,  are  some  of  the 
factors  that  must  be  considered  in  the  evalu¬ 
ation  .  Further ,  the  number  of  programs  to 
be  studied  is  limited  to  completed  contracts 
and  therefore  excludes  unsuccessful  propo¬ 
sals  and  incomplete  contracts.  However, 
because  eight  years  have  elapsed  since  the 
Vertol  risk  assessment  technique  was  first 
applied,  sufficient  programs  are  now  avail¬ 
able  for  a  comparative  evaluation. 

Seven  programs  were  studied,  including 
major  development,  research,  and  modifica¬ 
tion  programs;  aerospace  and  non-aerospace 
diversification  programs;  and  a  new  technical 
concept  research  and  development  program. 
The  comparison  was  made  relative  to  number, 
type,  and  severity  of  actual  problems  versus 
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potential  problems  identified  in  the  risk 
assessment,  and  final  "at  completion11  cost 
versus  the  predicted  probability  distribution. 
Results  of  this  comparison,  except  for  the 
non-aerospace  diversification,  indicated  a 
consistent  relationship  of  incurred  program 
cost  to  the  risk  described  in  the  original 
assessment  as  portrayed  by  potential  problem 
identification  and  cost  probability  distribu¬ 
tions.  The  non-aerospace  diversification 
analysis  proved  that  to  achieve  desired  risk 
assessment  results,  the  personnel  providing 
the  risk  data  must  have  sufficient  experience 
and  expertise  in  the  product  industry.  An 
example  of  the  aerospace  applications  was  the 
first  major  program  evaluated,  which  had  an 
excellent  correlation  with  the  risk  assess¬ 
ment.  The  actual  number  of  problems  was 
within  14%  of  the  number  projected.  The 
ratio  of  identified  versus  unidentified  pro¬ 
blems  that  occurred,  varied  by  only  5%  from 
the  prediction.  The  completion  cost  was 
equivalent  to  50%  probability  on  the  risk 
curve  These  program  comparisons  have 
provided  reasonable  substantiation  that  the 
Boeing  Vertol  risk  assessment  technique  has 
been  successful  in  quantitatively  predicting 
program  risk. 


CONCLUSIONS 


Successful  utilization  of  risk  assessment  at 
Boeing- Vertol  is  attributable  to  the  method 
by  which  the  three  essential  elements  of  the 
technology  are  accomplished. 

"0  Risk  Input  Data  -  A  minimum  of  easily 
obtainable  and  reliable  risk  information 
is  required  for  the  model.  The  primary 
data  is  formulated  using  potential  tech¬ 
nical  problems  that  are  relevant  to  the 
engineers  and  line  supervisors.  This 


enables  them  to  provide  reliable  proba¬ 
bilistic  data.  The  most  important  para¬ 
meter  is  couched  in  non -probabilistic 
terms  (most  likely  number  of  problems 
to  occur)  to  make  it  more  meaningful. 
The  problem  risk  data  is  first  reviewed 
in  detail  by  the  element  supervisor  and 
later  by  management.  Therefore,  the 
subjective  information  is  not  only  rea¬ 
sonable  to  obtain  but  is  relatively 
unbiased . 

2)  Computer  Model  -  The  model  realistically 
simulates  the  occurrence  of  risk  for  the 
proposed  program.  The  problem  simula¬ 
tion  portion  of  the  computer  model  dup¬ 
licates  the  ''real  life11  of  the  high  risk 
program  elements  by  increasing  the  cost 
and  allowing  the  schedule  to  slip  as 
problems  occur,  while  still  completing 
the  program  within  a  realistic  time  peri¬ 
od.  The  correction  factor  applied  by 
the  model  during  the  summation  of  pro¬ 
bability  curves,  properly  retains  total 
program  risk  by  negating  statistical 
con vergence. 

3)  Management  Acceptance  -  The  assess¬ 
ment  results  are  readily  amenable  to 
review  by  company  and  customer  man¬ 
agement.  The  risk  assessment  provides 
comprehensive  visibility  relative  to  the 
probability  of  achieving  the  cost  esti¬ 
mate,  the  range  of  possible  deviation, 
and  insight  into  the  causes  of  increased 
risk.  This  assessment  data  has  been 
proven  accurate  by  eight  years  of 
experience.  Therefore  the  risk  informa¬ 
tion  is  accepted  by  management  and 
utilized  with  confidence. 

The  Boeing  Vertol  quantitative  risk  assess¬ 
ment  technique  has  proven  to  be  a  uniquely 

successful  predictor  of  problem  risk. 
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ABSTRACT 


The  assessment  of  the  conditional  probabili¬ 
ties  of  events  is  useful  and  needed  for  fore¬ 
casting,  planning,  and  decision  making,  as  for 
example,  in  the  weapons  system  acquisition 
process.  The  difficulties  associated  with  the 
assessment  of  these  conditional  probabilities 
are  examined.  The  necessary  and  sufficient 
conditions  that  the  elicited  information  on 
conditional  probabilities  must  satisfy  are 
evaluated  against  actual  assessments  in 
several  different  controlled  settings.  A  high 
frequency  of  implicit  violations  of  the  proba¬ 
bility  calculus  was  observed.  The  consistency 
of  the  assessments  are  affected  by  the  causal/ 
diagnostic  and  positive/negative  relationship 
of  the  events.  Use  of  a  judgmental  aid  in  the 
form  of  a  joint  probability  table  reduces  the 
number  of  inconsistent  responses  significantly. 
Using  the  probability  axioms,  it  is  also  shown 
that  only  first  order  conditional  probabilities 
need  be  assessed,  as  such  higher  order  probabi¬ 
lities  are  robust  to  the  unconditional  and 
first  order  conditional  assessments. 


INTRODUCTION 


Conditional  probabilities  are  basic  input  to 
many  important  decision  models  that  are  used  to 
analyze  decisions  under  uncertainty  in  both  the 
private  and  public  sectors.  These  models  in¬ 
clude  decision  analysis  (e.g.,  [9],  [l5]), 
cross-impact  analysis  (e.g.,  [3],  [4],  [10]), 
long  term  probabilistic  forecasting  [l],  [7] 
fault  tree  analysis  [2],  [6],  and  several  other 
applications  of  statistical  decision  theory. 
Often,  if  not  usually,  the  conditional  probabi¬ 
lity  of  an  event  h  given  that  some  other  event 
B  is  known  to  have  occurred,  p(A|B),  can  only 
be  assessed  subjectively  by  an  "expert”.  The 
assessment  of  p(A|B)  is  intricate  and  subtle. 
While  there  are  numerous  experimental  and  field 
studies  concerned  with  the  assessment  of  mar- 
inal  event  probabilities  (e.g.,  [l],  [8], 

9]),  there  has  been  little  attention  paid  to 
the  elicitation  of  conditional  probabilities. 


The  elicitation  of  conditional  probabilities 
poses  additional  problems  that  are  not  en¬ 
countered  in  the  assessment  of  marginal,  proba¬ 
bilities  alone.  We  demonstrate  some  of  the 
resulting  difficulties  that  are  likely  to  be 
encountered  in  the  elicitation  of  conditional 
probability  assessments  and  suggest  ways  to 
improve  the  assessment  procedure.  The  diffi¬ 
culties  involve  the  implicit  violation  of  the 
probability  calculus  and  unmeaningfulness  of 
the  assessments  manifested  in  terms  of  their 
informativeness.  These  difficulties  are  shown 
to  be  affected  by  the  causal  versus  diagnostic, 
and  positive  versus  negative  relationship  of 
the  events  assessed.  We  also  show  that  employ¬ 
ment  of  a  joint  probability  table  (JPT)  as  a 
judgmental  aid  in  policing  assessments  improves 
their  consistency  and  meaningfulness  and  miti¬ 
gates  the  influence  of  the  causal/diagnostic 
nature  of  the  events  on  the  assessments. 
Moreover,  the  bounding  of  second  and  higher 
order  conditional  probabilities  (e.g., 
p(AjBnc),  p(A|  BA  C  TV D)  )  reduces  considerably 
the  number  and  complexity  of  the  conditional 
assessments  to  be  made  to  only  first  order  con¬ 
ditional  assessments,  p(A|B).  The  implications 
of  the  results  reported  should  be  useful  to  the 
decision  analyst,  policy  analyst,  and  planner 
in  obtaining  valid  and  meaningful  probabilistic 
input  from  experts  for  development  and  imple¬ 
mentation  of  forecasting  and  decision  models, 
such  as  in  weapon  system  acquisition. 

Our  principal  thesis  is  that  experts'  responses 
on  first  order  conditional  probabilities  alone, 
p(A|B)  (the  simplest  kind  of  conditional  proba¬ 
bility  assessment),  are  often  inconsistent  with 
the  simple  axioms  of  probability.  These  vio¬ 
lations  are  not  simply  e-level  errors  that  are 
expected  in  any  subjective  elicitation,  but  are 
errors  of  considerable  magnitude  that  result 
from  systematic  perceptual  and  cognitive  biases 
in  experts'  responses.  Moreover,  even  statis¬ 
tically  mature  experts  are  highly  susceptible 
to  these  errors.  The  interested  reader  should 
note  that  this  study  has  a  somewhat  different 
objective  than  the  widely  debated  phenomena 
that  the  subjects  do  not  revise  their  prior 
probabilities  according  to  Bayes'  rule  (e.g., 
[5],  [18],  [15  pp.  20-21]).  In  our  work 
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conditional  probability  is*a  basic  measurement 
that  cannot  be  computed  and  has  to  be  elicited 
from  the  subject.  In  order  to  convince  the 
reader  about  the  seriousness  of  these  viola¬ 
tions  and  provide  motivation  for  our  detailed 
experimental  investigations,  in  the  following 
section  we  report  some  representative  diffi¬ 
culties  that  were  encountered  in  assessing 
conditional  probabilities  in  an  actual  policy 
setting.  We  then  report  on  several  controlled 
experiments  involving  management  students  and 
business  executives  to  (1)  further  illuminate 
the  difficulties  experienced  in  assessing  con¬ 
ditional  probabilities,  (2)  better  understand 
the  assessment  process  and  several  factors 
which  affect  it,  and  (3)  evaluate  the  value  of 
the  JPT  as  a  judgment  aid. 


PROTOTYPICAL  EXAMPLES 
OF  INCONSISTENT  ASSESSMENTS 


In  a  study  to  determine  market  penetration  of 
solar  electric  energy  by  the  year  2000  several 
experts  from  governmental  agencies,  research 
laboratories,  and  utility  companies  were  inter¬ 
viewed  to  assess  the  likelihoods  of  future 
scenarios  [ 1 7 J .  Several  events  that  comprise 
a  future  energy  scenario,  such  as  restrictions 
on  oil-fired  capacity,  nuclear  slowdown, 
tighter  air  quality  standards,  commercial 
acceptance,  etc.,  were  identified.  Since  these 
events  are  interdependent,  the  marginal  and  the 
conditional  probabilities  of  occurrence  of 
these  events  were  required  as  an  input  to  the 
computation  of  the  scenario  probabilities. 

Below  we  provide  some  of  the  responses  of 
some  experts.  It  should  be  noted  that  for 
expositional  simplicity  we  are  using  the 
abbreviated  description  of  each  event.  In  the 
actual  application  these  events  were  precisely 
defined . * 

Response  Set  1: 

p(restrictions  on  oil  fired  capacity)  *  .9 
p(conservation)  =*  ,5 

p(conservation | restrictions  on  oil  fired 
capacity)  =  .7? 

Response  Set  2: 

p (nuclear  slowdown)  *  .3 
p(restrictions  on  oil  fired  capacity)  *  .9 
p ( res  trie tions  on  oil  fired  capacity j nuclear 
slowdown)  -  .6? 

Response  Set  3: 

p (incentives  to  solar  energy)  *  .6 
p (commercial  acceptance  of  solar  energy) 

*  .2 

p (commercial  acceptance | incentives)  -  .3 
p( incentives | commercial  acceptance)  »  .5? 
Each  of  the  conditional  probability  responses 
(question  mark  above)  is  inconsistent  with  some 
probability  axioms  as  will  be  shown  later. 
However,  even  many  decision  analysts,  manage¬ 
ment  scientists,  and  statisticians  agree  that 
the  above  responses  seem  reasonable.  It  is 
only  after  careful  analysis  and  introspection 
that  the  implicit  inconsistencies  become 
obvious . 


In  the  next  section  the  necessary  and  suffici¬ 
ent  conditions  that  the  elicited  conditional 
probability  must  satisfy  are  given. 


NECESSARY  AND  SUFFICIENT  CONDITIONS 


Suppose  A  and  B  are  two  events  of  interest  and 
the  marginal  probabilities  p(A)  and  p(B)  have 
been  assessed.  The  necessary  and  sufficient 
conditions  that  a  conditional  probability,  such 
as  p(B|A) ,  must  satisfy  in  order  to  be  consist¬ 
ent  with  the  probability  axioms  are  as  follows: 


p(B|A)  ^  1  (i) 

P<,|a)-M«  <2> 

P<B|«  >  ^AP«>  -  1  (J) 

p(B  j  A)  >_  0  (4) 


If  in  addition  to  p(B|A),  the  conditional 
probability  p(A|B)  is  also  assessed  then  these 
two  must  be  related. 

P(B|A)  p(AjB) 

p(B)  "  p(A)  (5) 

Thus,  if  p(B|A)  >  p (B) ,  then  p(A|B)  >  p(A). 

This  directional  consistence  is  obviously 
necessary  to  satisfy  condition  (5). 

Consider  the  prototypical  responses  of  the  ex¬ 
perts  given  in  section  2.  In  response  1  condi¬ 
tion  (2)  is  violated.  An  implication  of  this 
violation  is  that  p(restrictions  on  oil  fired 
capacity | conservation)  would  be  greater  than  1. 
In  response  2  condition  (3)  is  violated.  In 
response  3  conditions  (1)  -  (4)  are  satisfied 
but  condition  (5)  is  violated.  Although  it  is 
reasonable  to  expect  that  the  equality  in 
condition  (5)  may  not  be  exactly  satisfied  In  a 
subjective  assessment,  response  3  represents  a 
more  serious  violation  since  even  the  direction 
of  probability  revision  is  inconsistent. 

In  order  to  explore  the  nature  and  causes  of 
these  violations  several  detailed  controlled 
experimental  studies  were  undertaken,  some  of 
which  are  described  below.  These  studies 
dealt  only  with  the  assessment  of  pairwise 
(first  order)  conditional  probabilities  p (A | B> . 
The  problems  associated  with  the  more  complex 
assessment  of  higher  ordered  conditional  proba¬ 
bilities  >  e*3*>  p(A|Bf\C)  is  reserved  for  a 
later  section  of  the  paper. 


EMPIRICAL  INVESTIGATION 


4<1  Experiment  I:  Credit  Scoring  of  Loan 
Applications 


This  experiment  had  the  following  three  objec¬ 
tives:  (1)  to  examine  the  difficulties  in 

assessing  conditional  probabilities  in  terms  of 
the  frequency  of  consistency  violations 
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of  the  probability  calculus;  (2)  to  attempt  to 
understand  the  effect  of  the  causal/diagnostic 
and  positive/negative  relationships  of  the 
events  on  the  conditional  probability  assess¬ 
ments  and  the  frequency  of  violations;  (3)  to 
evaluate  the  use  of  the  JPT  as  a  judgmental  aid 
to  assist  the  assessor  in  policing  his  condi¬ 
tional  judgments  for  consistency  and  meaning¬ 
fulness  . 


4.11  Instrument  and  Task 


The  experimental  instrument  involved  a  case 
study  in  which  each  subject  assumed  the  role 
of  a  bank  lending  officer.  Given  the  base  rate 
probabilities  of  delinquency,  p(D),  and  the 
probability  of  having  either  a  bad  or  good 
credit  rating,  p(B)  or  p(G)  respectively, 
individuals  assessed  the  conditional  probabili¬ 
ties  p(D|B),  p(B|D),  and  p(D|G).  The  sets  of 
specific  values  for  p(D),  p(B),  and  p(G)  pro¬ 
vided  to  subject  are  given  in  Table  1.  The 
instrument  and  task  have  been  shown  to  possess 
a  variety  of  properties  that  make  it  well 
suited  for  systematically  investigating  human 
probabilistic  information  processing  behavior, 
while  providing  a  high  degree  of  subject 
interest  and  involvement  [l],  [12],  [l3]. 


Table  1 


4.12  Consistency  Violations 

Group  1  and  Group  4  questions  were  designed  to 
test  the  consistency  with  respect  to  conditions 
(2)  and  (3)  respectively.  Out  of  118  subjects, 
14  assumed  that  the  events  delinquency  and  bad 
credit  rating  are  statistically  independent. 

The  results  reported  are  for  104  subjects  who 
considered  credit  rating  to  have  some  impact  on 
delinquency.  In  Table  2,  the  consistency 
bounds  using  conditions  (1)  -  (4)  are  given  and 
the  number  of  violations  observed  for  each  of 
the  six  questions  in  Groups  1  and  4  are  report¬ 
ed.  It  is  clear  from  Table  2  that  a  large 
number  of  violations  were  observed.  Further, 

98  subjects  out  of  104  gave  inconsistent 
responses  on  at  least  one  out  of  these  six 
questions . 


Table  2 


In  group  1  questions,  the  evidence  of  bad 
credit  rating  enhances  the  likelihood  of 
delinquency,  A  possible  cause  of  violation  of 
condition  (2),  may  be  that  the  subjects  ignore 
the  base  rate  frequency  of  the  event  bad  cre¬ 
dit  rating  in  revising  the  probability  of 
delinquency.  As  the  base  rate  frequency  of 
bad  credit  rating  increases,  the  revision  in 
the  probability  of  delinquency  should  be 
smaller . 


Subjects  were  undergraduate  upper  division 
general  management  students  enrolled  in  a 
course  in  managerial  statistics  at  the  Krannert 
School  of  Management  at  Purdue  University.  The 
subjects  had  knowledge  of  probability  and  the 
concept  of  a  joint  probability  table  at  the 
time  they  participated  in  the  experiment. 

After  reading  the  situational  scenario,  sub¬ 
jects  first  specified  their  perceptions  of  the 
strength  of  the  relationships  between  a  credit 
rating  and  delinquency  on  a  5-point  scale  from 
very  weak  to  very  strong.  Then,  given  p(D) 
and  either  p(B)  or  p(G),  they  intuitively 
assessed  conditional  probabilities  such  as 
p(D|B),  p(B|D),  and  p(D|G)  for  15  questions 
partitioned  into  five  groupings.  They  next 
specified  how  they  arrived  at  their  assess¬ 
ments.  Then  given  a  JPT  with  the  marginal 
event  probabilities  included,  they  reassessed 
several  of  the  previous  questions.  Finally, 
they  rated  the  JPT  as  an  assessment  aid  on  a  7- 
point  scale  from  not  very  helpful  to  very 
help!  ill,  described  how  the  JPT  helped,  and  in- 
dlo.iic’d  in  what  way  tin*  table  aided  them  in 
revising  their  responses. 


In  group  4  questions,  the  evidence  of  good 
credit  rating  diminishes  the  probability  of 
delinquency.  Here  again,  the  subjects  seem  to 
ignore  base  rate  frequencies  of  p(D)  and  p(G) 
in  a  downward  revision  of  the  probability  of 
delinquency. 

The  questions  in  group  2  involved  specification 
of  p(BjD).  A  comparison  of  the  responses  for 
group  2  and  group  1  questions  revealed  that  the 
subjects  violated  directional  consistency.  To 
satisfy  directional  consistency,  p(B|D)  should 
be  greater  than  p(B)  since  p(D|B)  is  greater 
than  p(D).  The  number  of  violations  are  given 
below : 

p(D|B)  >  p(D) 

Q1  Q2  Q3 

P(B|D)  <  p(B)  Q4  33 

Q5  41 

Q6  58 

Notice  that  in  questions  4  to  6 ,  violation  of 
conditions  (1)  -  (4)  cannot  occur  since  the 
bounds  are  0-1.  It  was  further  observed  that  a 
large  number  of  subjects  had  no  consistent 
pattern  for  revision  as  the  base  rate  frequency 
of  bad  credit  rating  was  increased.  Intuitive¬ 
ly,  as  p (B)  increases  the  informativeness  of 
event  B^  decreases  and  therefore  the  relative 
increase  in  p (D ] B )  should  be  small.  In  the 
limiting  case  when  p(B)  =  1,  p(D|B)  =  p(D) . 

This  lack  of  pattern  in  the  direction  of 
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revision  was  also  observed  for  group  3  ques¬ 
tions  in  which  p(D)  =  .5.  This  suggests  that 
before  an  analyst  elicits  detailed  assessments, 
it  would  be  worthwhile  to  discuss  with  the  ex¬ 
pert  the  qualitative  impact  of  evidence  on  the 
event  for  which  the  conditional  probability  is 
being  assessed. 


4.13  Meaningfulness  Violations 


(Table  3a).  In  terms  of  imputed  informative¬ 
ness  the  probability  of  a  consistent  and  mean¬ 
ingful  response  is  about  0.4  (Table  3b). 
Alternately  stated,  the  average  number  of  such 
responses  per  question  was  only  35.6  (=  .4  x 
535  r  6)  or  30%  (=  .4  x  535  t  70S)  for  ques¬ 
tions  1-3  and  7  -  9;  38.1  (=  .44  x  260  f  3) 
or  32.0%  (=  .44  x  260  *  354)  for  questions  4  - 
6;  and  34.7  (*  .39  x  534  f  6)  or  0.29%  (=  . 39  x 
534  f  708)  for  questions  10  -  15. 


Assessments  that  are  not  meaningful  are  defined 
as  ones  that  either  imply  that  two  events,  say 
D  and  £,  are  statistically  independent  or 
uninformative  (when  they  should  not  be),  or 
whose  imputed  informativeness  is  contrary  to 
the  data  (i.e.,  in  a  direction  counter  to  the 
direction  of  revision  implied  by  the  data, 
assuming  no  "gaming",  which  was  verified  to  be 
the  case  in  this  experiment) . 


Statistical  independence.  In  this  experi¬ 
ment,  for  example,  a  statistically  independent 
response  would  imply  that  there  is  no  relation¬ 
ship  between  the  credit  rating  and  the  delin¬ 
quency  or  non-delinquency  of  the  loan  appli¬ 
cant,  i.e.,  the  likelihood  ratio 


L  -  P(B  jP)  _  i  , 

1  p (B  |  ND)  if  L2 


p  (c 

ND) 

p(G 

D) 

or  equivalently,  e.g.,  p(d|b)  =  p(D),  p(B|D)  = 

P(B)  . 


b.  Imputed  informativeness.  In  this  experi¬ 
ment,  responses  that  are  contrary  to  the  data 
result  in  assessments  which  reduce  the  proba¬ 
bility  of  delinquency  given  a  bad  credit  rat¬ 
ing,  and  enhance  the  probability  of  delinquency 
given  a  good  credit  rating,  from  the  prior 
probability  of  delinquency.  This  is  manifested 
by  implied  likelihood  ratios  of  L  »  p (B | D) / 
p(B|ND)  <  1  and  L  -  p (G|ND) /p(G| D)  <  1.  Dis¬ 
regarding  "gaming*  (e.g.-  disbelieving  the 
data  provided) ,  such  responses  presumably 
result  from  the  difficulty  of  making  condi¬ 
tional  assessments,  due  to  the  complex  inter¬ 
actions  involved  as  manifested  by  conditions 
(1)  -  (4).  More  precisely,  the  conditions  for 
a  meaningful  response  in  this  sense,  can  be 
derived  from  the  following  relationship,  which 
is  the  odds  representation  of  Bayes*  Theorem; 
e.g.. 


P(D|G)  _  p (D) 

p(ND|B)  p(ND) 


p(b|d) 


Hence,  if 


p(d|b) 


p (B | ND) 
P(D) 


(or,  more  simply,  if 


p(D 

B) 

p(B 

D) 

P  (B 

ND) 

Out 

of  , 

the 

14  < 

p(ND|B)  '  p (ND) 

>  P(D)),  then  the  likelihood  ratio 
>  1. 


tically  independent  to  most  questions),  the 
mean  number  of  statistically  independent 
responses  per  question  was  29.4  (~  25%) 


Table  3 


4.14  Factors  Influencing  Assessments 


The  conditional  probability  p(D'B),  can  be 
interpreted  as  a  revision  in  p(D)  given  the 
evidence  on  the  occurrence  of  the  event  B. 

If  the  relationship  between  £  and  £  is  per¬ 
ceived  to  be  strong,  a  greater  revision  is 
expected.  Two  factors  which  may  effect  the 
perceived  strength  of  the  relationship  between 
£  and  £  are  the  causal  versus  diagnostic  and 
the  positive  (confirming)  versus  negative 
(disconf irming)  relationship  of  these  events. 

Causal  versus  diagnostic.  Tver sky  and 
Kahneman  L 20 J  call  the  relationship  between  £ 
and  B  causal  if,  in  assessing  p (D ' B) ,  £  is  per¬ 
ceived  as  a  cause  of  the  occurrence  or  non¬ 
occurrence  of  £.  For  example,  in  our  experi¬ 
mental  setting,  where  the  event  of  interest  is 
delinquency  £,  if  the  event  £  is  a  bad  credit 
rating  then  the  event  £  is  a  causal  datum. 
Conversely,  the  event  £  is  called  a  diagnostic 
datum  if,  £  is  perceived  as  the  cause  of  B. 

Tversky  and  Kahneman  [20]  found  that  individu¬ 
als  perceive  a  causal  relation  as  more 
informative  than  a  diagnostic  relation.  In 
the  context  of  our  experiment  this,  for 
example,  implies  that 

If  B  "►  D  and  p(D)  =  C  p(B),  then 
p(D|B)  >  C  p  (B  |  D)  ; 

where  denotes  the  direction  of  causality, 

C  is  a  constant,  and  ^(D'B)  and  p(B|D)  are 
assessed  directly.  The  setting  of  our  case 
study  clearly  implies  that  the  event  £  is  a 
causal  datum.  If  a  causal  relation  is  per¬ 
ceived  more  informative  then,  for  example,  if 
in  question  1  a  subject  responds  p(DjB)  =  .2 
then  he  should  respond  p(B|D)  <  .6.  In  other 
words  the  value  of  p(D|B)  computed  from  p ( B | D ) 
should  be  smaller  than  the  assessed  value. 

This  indeed  was  the  case  in  our  experiment  as 
255  out  of  312  responses  showed  that  a  causal 
relation  caused  a  greater  revision  in  condi¬ 
tional  probability  than  a  diagnostic  relation. 


Q. 

p(d|b)>c  p(b|d) 

p (D | B) <C  p(B|D) 

Q1 

v/s 

Q4 

84 

8 

Q2 

v/s 

Q5 

91 

2 

Q3 

v/s 

Q6 

80 

7 
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Q- 

p(d|b)*c 

Q1 

V/S  Q4 

12 

Q2 

V/S  Q5 

11 

Q3 

V/S  Q6 

17 

The  above  table  shows  that  the  magnitude  of  the 
assessed  p(D|B)  was  consistently  greater  than 
that  imputed  from  p (B | D) .  The  causal  relation¬ 
ship  thus  led  to  more  substantive  revisions, 
and  therefore  higher  conditional  probability 
assignments  than  the  diagnostic  relationship. 
Theoretically,  there  should  be  no  difference. 

This  was  further  substantiated  in  a  separate 
experiment  in  which  36  different  management 
students  assessed  p(D|B)  and  p(B|D)  on  three 
questions,  given  the  same  probabilities  of  .2, 
.5,  and  .8  for  p(D)  and  p(B).  In  this  case, 
the  assessments  could  be  compared  directly 
since  p(D|B)  should  be  equal  to  p(B|D).  It  was 
found  that  more  than  twice  as  many  of  the 
responses  indicated  p (D | B)  >  p (B [ D )  than 
p(D|B)  <  p(BjD).2 

A  causal  relation  leads  to  a  more  substantive 
revision  in  conditional  probability.  Thus, 
with  a  causal  relationship,  more  violations 
than  expected  may  occur.  Individuals,  however, 
find  it  easier  to  assess  p (D  |  B)  if  _B  is  causal 
and  have  more  confidence  in  their  assessments 
[20].  This  could  represent  a  paradox  for  an 
analyst  since  a  more  "meaningful"  assessment 
may  also  give  more  inconsistencies.  This 
observation  further  strengthens  our  argument 
for  the  use  of  an  assessment  aid  such  as  a  JPT. 

Positive  versus  negative.  A  positive  (nega¬ 
tive)  relationship  between  events  _B  and  D  im¬ 
plies  that  knowledge  of  13  occurring  should 
increase  (decrease)  the  probability  of  JD 
occurring.  Individuals  may,  however,  perceive 
a  positive  relation  as  more  informative  than  a 
negative  relation  [ll] ,  [l2],  [l3].  In  our 
experiment  this,  for  example,  inplies  that 

P(D|B)  >  P.W  -  p(DjC)  :  P(C) 

1  P  (B) 

where  £  denotes  a  good  credit  rating,  and 
p(D|B)  and  p(D|G)  are  directly  assessed  by  an 
individual.  Again,  this  would  imply  that  a 
positive  relation  between  events  would  lead  to 
more  violations. 

As  shown  in  Table  4  the  value  of  p(D|B)  was 
also  consistently  greater  than  that  imputed 
from  p(D|G) .  A  positive  relationship  between 
events  thus  also  appeared  to  be  more  informa¬ 
tive  than  a  negative  relationship.  This  too 
was  further  corroborated  in  another  experiment  . 
in  which  17  business  executives  in  a  forecast¬ 
ing  seminar  assessed  p(d|b)  and  p(D|G),  given 
probabilities  on  p(D)  of  .2,  .5,  and  .8  and  a 
probability  of  .5  on  p(B)  and  p(G). 


Table  4 


4.15  JPT  as  an  Assessment  Aid 


The  reduction  in  consistency  violations  result¬ 
ing  from  using  a  JPT  as  an  assessment  aid  are 
shown  in  Table  5 .  There  was  a  marked  improve¬ 
ment  in  consistency.  The  overall  frequency  of 
violations  of  conditions  (1)  -  (4)  was  reduced 
from  62%  to  15%.  Directional  violations  were 
also  markedly  reduced.  Before  using  the  JPT, 

58  subjects  had  specified  p(D  B)  >  p(D)  and 
p(B|D)  <  p(B).  But,  after  using  the  JPT  only  5 
subjects  violated  directional  consistency, 
almost  a  12-fold  improvement. 

In  terms  of  meaningfulness,  the  average  number 
of  statistically  independent  responses  per 
question  was  reduced  only  slightly  from  29 
(**  24.6%  of  all  responses)  to  24.6  (=  20.8%). 

In  fact,  it  was  just  as  likely  for  an  individu¬ 
al  to  switch  from  an  independent  to  a  non- 
independent  response  with  the  use  of  the  JPT, 
as  conversely.  Thus,  the  JPT  had  little  effect 
on  reducing  the  number  of  such  responses. 
However,  the  frequency  of  meaningful  responses 
in  terms  of  imputed  informativeness  (which  was 
also  consistent  with  conditions  (1)  -  (4))  was 
increased  from  19.6  (*  11.6%  of  all  responses) 
to  52.8  (■  44.7%)  per  question,  a  270%  improve¬ 
ment.  The  value  of  the  JPT  is  further  high¬ 
lighted  by  about  a  7-fold  increase  in  the 
number  of  individuals  whose  responses  obeyed 
all  consistency  and  meaningfulness  conditions 
(e.g.,  in  questions  3  and  6  the  increase  was 
from  9  to  66) . 

The  causality /diagnosticity  effect  was  also 
mitigated  significantly  by  the  use  of  the  JPT. 

To  illustrate,  before  using  the  JPT,  out  of  104 
responses  to  questions  3  and  6,  80  responded 
p(DjB)  >  C  p(B|D),  7  responded  p(D[B)  < 

C  p(B|D) ,  and  17  responded  p(D|B)  =  C  p(B[D). 
Using  the  JPT,  14  responded  p(D[B)  >  C  p (B | D) , 

6  responded  p(D|B)  <  C  p(B|D),  and  84  responded 
p(D|B)  =  C  p (B | D) .  A  large  number  of  subjects 
indicated  that  they  found  the  JPT  to  be  a 
helpful  assessment  aid. 


Table  5 


4.2  Experiment  II:  Lincoln  Securities 
Analysts,  Inc. 

This  experiment  differed  from  the  previous  one 
in  the  following  respect:  (1)  the  context  in¬ 
volved  an  actual  problem  situation;  (2)  both 
marginal  as  well  as  conditional  probabilities 
were  assessed.  The  objective  was  to  determine 
if  there  were  differences  in  the  results 
between  this  experiment  and  the  previous  one, 
due  to  the  problem  situation  and  assessing  the 
marginal  events  as  opposed  to  having  them 
provided . 
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The  task  was  to  predict  the  following  events: 

1.  The  likelihood  that  the  Dow  Jones  Industri¬ 
al  Index  (DJI)  as  of  July  1,  1979  would  be 
in  a  specified  range,  i.e.,  p(DJI  <  800), 

p (800  £  DJI  <  900),  p (DJI  £  900). 

2.  The  likelihood  that  real  GNP  growth  for  the 
first  6  months  of  1979  would  have  the  rate 
of  growth  specified,  i.e.,  p(GNP  <  2%), 

p (2%  £  GNP  <  4%),  p(4%  £  GNP  <  6%), 
p (GNP  £  6%) . 

3.  The  likelihood  that  the  DJI  as  of  July  1, 
1979  would  be  in  a  specified  range  given 
real  GNP  growth  for  the  first  6  months  of 
1979  would  have  the  specified  rate  of 
growth . 

In  certain  cases,  the  likelihood  that  GNP 
growth  would  have  the  rate  of  growth  specified 
given  the  DJI  would  be  in  a  specified  range  was 
also  assessed. 

Eighty-nine  management  students  participated. 

Of  these,  40  were  discarded  —  3  did  not  have 
their  various  event  probabilities  sum  to  1, 
although  instructed  to  do  so;  8  gave  probabi¬ 
lity  responses  that  were  >  1  or  <0;  29  gave 
conditional  probability  responses  that  were 
strictly  independent. 


Results 


The  frequency  of  consistency  violations  was 
less  in  this  experiment  than  in  the  previous 
one,  because  the  marginal  probabilities 
assessed  by  subjects  often  resulted  in  loose 
consistency  bounds  for  the  conditional  proba¬ 
bilities  (contrary  to  the  previous  experiment, 
a  separate  bound  existed  for  each  individual 
based  on  his  marginal  probability  assessments). 

Use  of  the  JPT  reduced  the  frequency  of  viola¬ 
tions  in  half,  from  40  to  20  out  of  196 
responses  examined.  Directional  violations 
were  also  reduced  significantly.  The  frequency 
of  meaningful  responses  increased  from  34%  to 
51%.  The  JPT  had  little  effect  on  altering 
individuals’  marginal  probability  assessments, 
as  reflected  below: 


Changed  Marginals 
Yes  No 


Changed 

Conditionals 

Yes 

18% 

72% 

90% 

No 

0% 

10% 

10% 

18% 

82% 

100% 

This  table  shows  that  the  subjects  tend  to  have 
more  confidence  in  their  assessment  of  marginal 
probabilities  and  are  more  likely  to  revise 


their  conditional  probability  assessments  if  an 
assessment  aid  such  as  a  JPT  is  offered. 


4.3  Discussion 


The  results  reported  in  the  experiments  under¬ 
score  the  complexities  and  concomitant  diffi¬ 
culties  associated  with  assessing  simple  first 
order  conditional  probabilities.  These  diffi¬ 
culties  were  manifested  by  a  high  frequency  of 
consistency  violations  of  the  probability  cal¬ 
culus  as  well  as  assessments  that  were  unmean¬ 
ingful.  Such  factors  as  the  causal/diagnostic 
and  positive/negative  relationship  between 
events  also  affected  the  nature  and  quality  of 
the  responses.3 

The  use  of  a  JPT  resulted  in  a  dramatic  im¬ 
provement  in  consistency  and  meaningfulness  of 
the  conditional  probability  assessments  and 
mitigated  the  causality/diagnos ticity  "effect”. 
The  results  of  the  post-experimental  question¬ 
naires  further  indicated  that  the  subjects, 
particularly  those  who  initially  violated  con¬ 
ditions  (1)  -  (4),  found  the  joint  probability 
table  highly  useful  as  an  assessment  aid. 
Moreover,  many  of  those  who  were  consistent, 
in  fact,  developed  their  own  JPT  in  making 
their  initial  assessments.  Most  subjects  in 
these  experiments  were  familiar  with  probabi¬ 
lity  concepts  and  JPT's  and  therefore  infer¬ 
ences  must  be  limited  to  such  groups. 

It  may  be  argued  that  to  avoid  the  problem  of 
consistency,  an  analyst  could  compute  the  bounds 
using  conditions  (1)  -  (4)  and  require  the 
decision  maker  (subject  or  expert)  to  supply 
the  estimates  of  conditional  probability  within 
the  computed  bounds .  This  would  avoid  the 
necessity  of  an  assessment  aid  such  as  a  JPT. 

We,  however,  feel  that  the  violation  of  consist¬ 
ency  is  merely  a  symptom  for  cognitive  biases 
and  lack  of  understanding  of  the  concept  of 
conditional  probability.  Thus,  even  in  those 
cases  where  consistency  violations  cannot  occur 
e.g.,  when  bounds  are  between  0  and  1,  an 
assessment  aid  could  improve  the  reliability  of 
the  assessments.  This  question  of  reliability, 
however,  needs  to  be  examined  in  situations 
where  external  observations  are  available 
against  which  subjects'  assessments  can  be 
evaluated  (e.g.,  [l4],  [22]). 

HIGHER  ORDER  CONDITIONAL  PROBABILITIES 


The  assessment  of  higher  order  conditional 
probabilities  is  even  more  complex  a  task  than 
the  assessment  of  simple  pairwise  conditional 
probabilities.  This  is  because  a  conditional 
probability  such  as  p(A!BnC)  must  satisfy  a 
larger  number  of  constraints.  For  example,  if 
p  (A)  ,  p  (B) ,  p(D),  p(A|B),  p(A|C),  and  p  (B  |  C) 
are  assessed,  then;  the  constraints  that 
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p(a|b  Oc)  must  satisfy  are  as  follows: 


( 


P  (A  BA  C)  >_ 
p (A  BAC)  > 

p(A|BftC)  > 

P  (A  j  B/\  C)  >_ 


P  (A 
P  (A 
p(A 
P  (A 


BAC)  <_ 
BAC)  < 
BAC)  £ 
B  A  C)  < 


0 

[p  (A 
p(B 
[p  (A 
P(B 
[p  (A 
p(B 
P  (A 
P  (A 
P(B 


B)  p (B) 

C)  p(C) 

B)  p(B) 

C)  p(C) 

C)  p(c) 
c)  p(C) 

B)  p (B)  p (B 
c)  p(C)  p(B 

C)  p (C)  p (B 


P(A|c)  p(C)  -  p(A) j 
P(b|C)  p (C)  -  p (B) ] 
P (B | C)  p (C)  -  p(C)] 
C)  P(C) 

C)  p(c) 

c)  p(C) 


[l  -  p (A)  -  p (B)  -  p (C)  +  p(A|B) 
P(B)  +  P(A|C)  p(C)  +  p(B|C)  p(C)] 
P (B  j  C)  p(C) 


Besides,  the  problem  of  consistency  violation 
due  to  a  larger  number  of  constraints,  subjects 
find  it  difficult  to  consider  the  interaction 
between  the  three  events.  To  lend  further  sup¬ 
port  that  the  mathematical  definition  of  the 
conditional  probability  and  the  psychological 
notion  that  people  employ  in  supplying  these 
probabilities  are  often  inconsistent,  consider 
the  responses  of  a  group  of  graduate  students 
in  Business  Administration  (11  students)  that 
were  given  the  following  information: 

p(A|B)  >  p (A)  and  p(A|C)  >  p (A) . 

They  were  then  asked  to  estimate  the  lowest 
value  of  the  conditional  probability  of 
event  A  given  both  £  and  £  occur;  i.e.  , 
P(A|bOC).  Many  replied  it  has  to  be  the 
max  {p (A | B) ,  p(A| C) } .  Some  said  it  could  be 
the  min  { p (A | B) ,  p (A | C ) } .  None  agreed  that  it 
could  be  less  than  p(A).  The  reader  can  easily 
verify  that  if  p(A)  =  p(B)  =  p(C)  -  .1  and 
p(A|B)  =  .2,  p(A|c)  -  .3,  p(B|C)  *=  .1  then  it 
is  possible  to  have  p(A|BHC)  =  0.  In  the 
special  case  when  the  probability  that  neither 
of  the  events  will  occur  is  0,  it  is  indeed 
true  that  p(a|bAc)  £  min  {p(A|  B) ,  p(A|c)}. 

We  now  show  that  such  higher  order  conditional 
probability  assessments,  which  further  compli¬ 
cate  the  assessment  task  considerably,  can  be 
largely  circumvented  by  appropriate  bounding. 

We  will  discuss  the  strategy  of  bounding  higher 
order  probabilities  in  the  context  of  an 
example.  Our  results,  however,  equally  apply 
to  other  decision  contexts. 


Consider  n  binary  events,  where  each  one  of  the 
events  e 1 ther  occurs  or  does  not  occur.  Then 
there  exist  2  combinations  of  occurrence  or 
non-occurrence  of  these  sequences  of  events. 

Each  combination  is  defined  as  a  scenario.  For 
example,  if  n  =  3,  one  scenario  is  that  the 
event  1  occurs  and  the  events  2  and  3  do  not 
occur;  yet  another  scenario  is  that  event  1 
does  not  occur  and  events  2  and  3  occur,  and  so 
on.  The  assessment  of  the  probabilities  of 
each  of^the  2  scenarios  would  minimally  re¬ 
quire  2  -  1  judgments  from  the  experts.  An 

approach  proposed  in  Sarin  [l6]  is  to  elicit  n 
marginal  probabilities  plus  n(n-l)/2  conditional 
probabilities  of  the  pairs  of  events  (i.e., 
first  order  conditional  probabilities)  .  This 


information,  which  is  considerably  less  than 
2  -1  probability  judgments  if  n  is  even 
moderately  large  is  then  used  to  bound  the 
higher  ordered  conditional  (or  joint)  probabi¬ 
lities.  That  is,  if  there  are  four  events,  we 
need  to  know  p(E. ) ,  i  =  1,  . . . ,  4;  p(E  0  E  ) 

i  =  1 . 3;  jx>  i;  p(E  <\E  rtt),  ix=  1^  2; 

k  >  j  >  i;  and  p(E,A  E.  I\  E  (^E.)  to  completely 
specify  the  probability  of  each  of  the  16 
scenarios.  However,  using  Sarin’s  approach 
[l6],  elicitation  of  only  four  marginal  proba¬ 
bilities,  six  first  order  conditional  (or 
joint)  probabilities  of  the  pairs  of  events 
need  be  elicited.  These,  as  it  turns  out, 
place  tight  bounds  on  the  higher  ordered  joint 
probabilities . 


It  can  be  shown  that  the  theoretical  maximum 
difference  between  the  upper  and  lower  bound  of 
p(E  A  E  AE  )  is  1/2.  That  is,  no  matter 
what  vaiuesnwe  choose  for  p(E  )  and  p(E.A  E  ) 
in  the  four  event  case  above, ^the  value^of  - 
p(E.O  E„/1E  qE^)  must  lie  within  a  range  of 
0.1^5.  We  have  conducted  a  simulation  study  in 
which  different  values  of  marginal  and  joint 
probabilities  of  the  pairs  of  events  were 
randomly  generated  from  uniform  distributions. 
The  random  assignments  to  joint  probabilities 
were  made  using  the  consistency  bounds.  The 
difference  in  the  upper  and  lower  bounds  for 
the  higher  ordered  joint  probabilities  was 
computed.  It  was  found  that,  for  90%  of  the 
problems  generated  (100  problems  were  generated 
for  each  of  the  3  events  and  4  events  cases), 
the  difference  in  the  upper  and  lower  bounds 
for  p(E  A  E  A  E  )  was  <_  .04  and  for 
p(E,  AE  AE  <AE  )  was  <_  .006.  The  mean  and 
standard  deviation  of  the  difference  in  the 
upper  and  lower  bounds  for  p(E  A  E  A  E  )  were 
.015  and  .017  and  that  for  p(E  A  E^A  E^A  E  ) 
were  .003  and  .004.  These  results^ind^catl 
that,  practically  speaking,  only  the  marginal 
and  pairwise  conditional  (or  joint)  probabili¬ 
ties  need  to  be  elicited.  The  analyst  or  the 
expert  can  then  choose  the  values  for  the 
higher  ordered  joint  (or  conditional)  proba¬ 
bilities  within  the  specified  bound,  with 
apparently  little  adverse  significance. 


CONCLUSIONS  AND  FURTHER  RESEARCH  SUGGESTIONS 


The  assessments  of  conditional  probability 
seem  so  deceptively  simple  that  the  difficulties 
associated  with  these  assessments  are  largely 
ignored  in  the  literature.  Conditional  proba¬ 
bilities  are,  however,  basic  input  to  many 
forecasting  and  decision  models.  We  have  shown 
that  a  large  number  of  consistency  violations 
of  the  probability  calculus  occur  in  a  direct 
assessment  of  conditional  probabilities.  These  ■ 
violations  are  not  due  to  small  random  errors 
that  are  expected  in  any  subjective  elicitation, 
but  are  errors  of  considerable  magnitude  that 
result  from  systematic  perceptual  and  cognitive 
biases  in  subjects1  responses.  We  have  found 
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that  the  conditional  probability  assignments 
are  considerably  larger  if  the  relationship 
between  the  events  is  perceived  to  be  causal 
rather  than  diagnostic.  Similarly,  a  positive 
relationship  between  events  leads  to  a  larger 
assignment  of  conditional  probability  than  a 
negative  relationship.  Normatively,  the 
assignments  of  conditional  probability  should 
not  be  influenced  by  the  causal  or  diagnostic 
and  positive  or  negative  relationships  of  the 
events . 

Our  results  clearly  show  that  the  use  of  an 
assessment  aid  such  as  Joint  Probability  Table 
results  in  a  dramatic  improvement  in  consis¬ 
tency  as  well  as  mitigates  the  causality/ 
diagnosticity  and  positive/negative  effects. 

We  have  also  shown  that  the  even  more  complex 
task  of  assessing  higher  order  probabilities 
can  be  largely  circumvented  by  appropriate 
bounding.  This  simplifies  the  assessment  task 
considerably  in  many  real  forecasting  and 
decision  situations  by  reducing  the  number  and 
level  of  difficulty  of  the  assessments. 

To  further  understand  how  people  process  in¬ 
formation  in  assigning  conditional  probability 
a  psychological  experiment  would  be  desirable. 
The  merits  of  eliciting  joint  as  opposed  to 
conditional  probabilities,  as  well  as  using 
assessment  aids  other  than  a  JPT  also  need  to 
be  investigated.  Similarly,  the  question  of 
reliability  of  assessments  remains  to  be 
studied.  We  believe  that  if  an  analyst 
interacts  with  the  expert  on  a  one-to-one 
basis,  as  should  be  the  case  in  a  real  appli¬ 
cation,  the  effectiveness  of  an  assessment  aid 
such  as  a  JPT  will  further  increase.  More 
experimental  and  field  studies  are,  however, 
needed  to  answer  these  questions.  In  the 
interim  we  propose  that  an  analyst  should  make 
the  decision  maker  aware  of  the  consistency 
conditions  and  discuss  with  him  qualitatively 
the  Impact  of  evidence  on  the  event  for  which 
conditional  probability  is  being  assessed.  The 
analyst  should  also  use  an  assessment  aid  such 
as  JPT  in  the  elicitation  process.  The  real- 
life  and  classroom  experiments  that  we  have 
undertaken  over  the  past  few  years  provide 
overwhelming  evidence  that  even  in  relatively 
simple  situations  an  unaided  assessment  of 
conditional  probabilities  leads  to  serious 
errors  of  considerable  magnitude. 


FOOTNOTES 

1 

In  weapons  acquisition,  such  events  that  would 
comprise  a  weapons  system  acquisition  scenario 
might  broadly  be,  cost  of  the  weapons  system, 
time  required  for  system  development,  capabi¬ 
lity  of  achieveing  specified  performance 
characteristics,  and  so  forth  [7]. 


Sometimes  it  is  possible  to  reduce  the  discre¬ 
pancy  between  causal  and  diagnostic  responses 
of  the  probabilities  by  a  careful  structuring 
of  the  questions.  This  would  be  the  case,  for 
example,  in  Response  3  of  the  solar  energy 
example.  We  could  ask:  what  is  the  chance 
that  incentives  had  been  given  to  solar  energy 
given  that  commercial  acceptance  is  observed. 
Now,  we  can  expect  p(l|C)  >  p(I)  .  But,  our 
results  still  show  that  the  relative  magnitude 
of  p(D|B)  is  higher  than  p (B  J  D)  even  if  care¬ 
ful  questioning  avoids  temporal  or  other 
consideration. 

.  ^ 

The  following  standardized  regression  model 
was  developed  to  predict  the  %  of  individuals 
responding  consistently  to  conditions  (1)  - 
(4)  (standard  errors  in  parentheses) : 

%NV  =■  -.  45P  -.  46T  -  .19C 

(-.15)  (-.44)  (.16) 

where 

NV  «  /0  of  individuals  responding  consistent¬ 
ly; 

P  *  positive  (-  1)  or  negative  (*=  0) 
relation  between  events; 

T  =  range  of  tightness  of  consistency 
bound  (=  upper  -  lower  bound) ; 

C  =  causal  (=  1)  or  diagnostic  (-  0) 
relationship  between  events. 

_2 

R  turned  out  to  be  equal  to  .50.  The  %-age 
of  consistency  violations  increased  with  a 
positive  relationship  between  events  and 
tighter  bounds.  Causality  had  only  a  small 
negative  effect  on  consistency. 
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Table  1 


Credit  Loan  Application  Assessments 

D  ■  event,  delinquent 

B  *  event,  bad  credit  rating 

G  -  event,  goo‘d  credit  rating 

p(’)  ■  probability  of  a  given  event 


Group  Questions 
G  1  Q  1 

Q  2 
Q  3* 

G  2  Q  A 

Q  5 

Q  6* 

G  3  Q  7 

Q  8 
Q  9* 

G  4  qio* 

Qll 

Q12 

G  5  Q13 

Q14 
Q15* 


?(D)  P (5)  p (G) 

.1  .3 

i 

•  X  .3 

.5 
.9 

.5  .3 

•  5 

•  9 

•9  .3 

.5 
.9 

•X  .1 

I’  -5 


Consistency 
Response  Range  Bounds 

p (D | B)  o  -  .33 

0  -  .20 

*  0  -  .11 

P(BID)  0  -  1.00 

0  -  1.00 

o  -  1.00 

p CD i B)  o  -  l.oo 

0-1.00 
J  .44  -  .55 

p  CD i G>  .66-1.00 

.so  -  l.oo 

*  .88  -  1.00 

p CD | G)  o  -  1.00 

0  -  2.20 

0  —  0.143 


♦Responses  to  these  questions  were  also  obtained 
a  joint  probability  table  as  a  decision  aid. 


subsequently  using 


Note  that  the  following  questions  are  equivalent: 
Q  1  and  Q15 
Q  2  and  Q14 
0  3  and  Qll. 


245 


Table  2 


Frequency  of  Consistency  Violation 


(total  responses 

<r 

o 

H 

Group 

Questions 

PCD) 

p(B) 

P(G) 

Response 

Consistency 

Number  of 

Bounds 

Violations 

1 

1 

.1. 

.3 

p(D/B) 

0  -  .33 

31 

2 

.1, 

.5 

p(D/3) 

0  -  .  2 

70 

3 

.1 

.9 

p(D/B) 

0  -  .11 

69 

4 

10 

.9 

.2 

?(D/G) 

.66  -  1 

58 

11 

.9 

.5 

p(D/G) 

.80  -  1 

87 

12 

.9 

.9 

p(D/G) 

.88  -  l 

78 

Table  3 

Meaningfulness  of  Assessments 

a'  ^HSber  of  Statistically  Independent  Responses  (Condition  6) 


Question  No. 

Re  sj 

>onse  (n  *  Us  responses  per  question) 

P(D|B) 

P(B/D) 

P(DIG) 

1 

21 

2 

18 

3 

31 

4 

32 

5 

27 

6 

35 

7 

20 

8 

59 

9 

24 

10 

18 

11 

16 

12 

33 

13 

43 

14 

27 

15 

37 

Total 

173 

94 

174 

Mean  \ 

28.83 

31.33 

29.00 

"  '  U's  ’ '’"i'""**5*  P«v  question,  including  the  14  who  responded  as 
statistically  independent  to  most  questions. 
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(TABLE  3  CON'T) 

t 

Joint  Probability  of  Consistent /Inconsistent  (Conditions  (1)  -  (4))  and 

Meaning ful/ Unmeaning ful  (Condition  (7))  Responses 

Q  1,  2,  3;  and  Q  7,  8,  9  (n  »  108  x  6  =*  708  responses)* 


Incons istent 

Consistent 

Meaningful  p(D|B)  >  p(D) 

.44 

C 3 ) 

.84 

Unmeaningful  p(D|B)  <  p (D) 

.02 

.14 

.16 

.46 

.54 

1.00 

*173  responses  were  statistically  independent;  these  were  not  included  in 
the  above  table,  which  was  thus  based  on  535  responses. 


5,  6  (n  «  118  x  3  *  354  responses)* 

Inconsistent 

Consistent 

Meaningful  p(d|b)  >  p(D) 

0 

© 

.44 

Unmeaningful  p(D|B)  <  p(D) 

0 

.56  | 

.56 

0 

1.00 

1.00 

*94  responses  were  statistically  independent;  these  were  not  included 
in  the  above  table,  which  was  thus  based  on  260  responses. 

-  15  (n  38  118  x  6  =  708  responses)* 

Inconsistent 

Consistent 

Unmeaningful  p(D|G)  >  p(D) 

.17 

.19 

.36 

Meaningful  p(D|G)  <  p(D) 

.25 

© 

.64  ' 

.42 

1 

Ln 

OO 

1.00 

*174  responses  were  statistically  independent; 

these  were  not 

included 

in  the  above  table,  which  was  thus  based  on  534  responses. 


Question 

Q1  v/s  Q15 
Q2  v/s  Q14 
Q3  v/s  Q13 


Table  4 

Positive  v/s  Negative  Causality 

p(D|B)  >cou?uted  p(D  |  B)  <coir.puted  p  (D  I  B)»conputed 

value  fron  p(D|C)  value  from  p(DjG)  value  from  p(D|G) 


92 

94 

82 


9 

5 

13 
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Table  5 


Effect  of  Decision  Aid  (Joint  Probability  Table) 
on'  Consistency  Violations 


Question 

Sample 

Freouencv  of 

Violations 

Number 

(Size  (n') 

Before 

After 

3 

104 

69  (662) 

4  (  4%) 

rr* 

104  | 

1  0 

~ ol 

9 

104 

78  (75%) 

9  (  9%) 

10 

104 

58  (56%) 

20  (19%) 

15 

104 

51  (49%) 

30  (29%) 

Total** 

416 

256  (62';) 

62  (is:;) 

* In  this  question,  the  counts  could  not  be  violated  since 
ranged  over  the  entire  probability  space,  from  0  co  1. 

**Does  not  include  06,  since  bounds  could  not  be  violated. 
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A  PROCEDURE  FOR  COST-RISK  ASSESSMENT 

Dr.  Gerald  R.  McNIchols,  Management  Consulting  &  Research,  Inc. 
5203  Leesburg  Pike,  Suite  608,  Falls  Church,  VA  22041 


ABSTRACT 


An  emerging  weapon  system  concept  is 
typically  judged  by  its  military  worth.  The 
military  worth  of  a  weapon  system  is  often 
evaluated  by  several  criteria: 

•  system  performance 

•  life  cycle  cost 

•  developmental  risk. 

Recent  revisions  to  Department  of  Defense 
(DOD)  Directives  and  Instructions  (e.g. 

5000.1,  5000.2,  and  5000.39)  stress  the  impor¬ 
tance  of  performing  an  adequate  risk  analysis. 

This  paper  briefly  describes  one  key  problem 
of  risk  assessment.  A  risk  assessment  must 
include  not  only  the  likelihood  of  success, 
but  also  the  consequences  of  failure  in 
quantitative,  measured  terms,  such  as  dollars. 

Management  Consulting  &  Research,  Inc.  (MCR) 
has  been  involved  in  weapon  system  risk  anal¬ 
ysis  for  some  time.  This  particular  paper 
describes  a  simple  procedure  for  performing 
a  risk  analysis  and  integrating  cost  conse¬ 
quences  into  the  model.  This  simple  method¬ 
ology,  called  "cost-risk  assessment"  will  be 
described.  We  will  then  examine  its  u'se  in 
analyzing  the  effect  of  competition.  The 
mathematical  formulas  are  not  included  in 
this  paper,  but  are  available  from  the  author 
or  references  (particularly  3). 


INTRODUCTION 


Within  the  past  year  several  events  have 
occured  to  reinforce  the  increasing  emphasis 
on  risk  and  resource  analysis  problems  within 
the  Department  of  Defense.  Specifically  DoD 
Directive  5000.1,  "Major  Systems  Acquisition," 
was  updated  on  March  19,  1980  to  fully  imple¬ 
ment  the  concepts  and  provisions  of  Office  of 
Management  and  Budget  Circular  A-109;  DoD 
Instruction  5000.2,  "Major  System  Acquisition 
Procedures,"  was  also  updated  on  March  19,  1980 
to  include  procedures  for  acquisition  planr 
ning;  DoD  Directive  5000.39,  "Acquisition  and 
Management  of  Integrated  Logistic  Support  for 
Systems  and  Equipment,"  was  issued  January 
17,  1980  to  establish  policy  and  responsi¬ 
bilities  for  Integrated  Logistic  Support  (ILS) 
In  addition  untold  studies  of  particular 
mission  areas  or  weapon  systems  have  examined 
mid-mix  design  alternatives,  assessment  of 
risks  bases  on  sensitivity  to  uncertainty  in 
key  design  and  logistic  parameters,  and  life 
cycle  cost  considerations. 


In  evaluating  contractor  proposals  for  specific 
responses  (under  the  A-109  philosophy)  to  DoD 
Mission  Element  Needs  Statements,  we  are  begin¬ 
ning  to  observe  evaluation  criteria  of  a 
system1 s  military  worth  such  as: 

•  system  performance 

•  system  life  cycle  cost 

•  developmental  risk. 

Typically  the  developmental  risk  evaluation 
factor  assesses  the  degree  to  which  a  contrac¬ 
tor's  proposed  system  is  likely  to  achieve  its 
predicted  performance  within  the  predicted  cost 
and  schedule  goals.  In  evaluating  or  in  con¬ 
ducting  any  risk  analysis,  it  is  essential  to 
consider : 

•  Risk  Assessment:  Identify  the  degree 
of  technical  risk  with  respect  to 
realism,  soundness,  and  credibility 
(i.e.,  the  technical  feasibility  as 
well  as  the  "contractor  bias  problem). 
Answer  questions  such  as:  Are  there 
low-risk  alternatives  to  high-risk 
components?  Row  much  would  perfor¬ 
mance  suffer?  What  additional  cost 

is  incurred  for  continuing  multiple 
alternatives? 

•  Risk  Management:  Develop  a  plan  for 
managing  all  types  of  risk  (risk 
minimization  plan)  as  a  function  of 
time  (i.e.  Acquisition  Milestone  I, 

II,  and  III).  The  role  of  quality 
assurance,  hedges  against  new  tech¬ 
nology  failure,  etc.  are  considered 
here. 

•  Risk  Demonstration:  A  test  and  eval¬ 
uation  demonstration  plan  should 
allow  early  identification  of  risks 
and  the  steps  required  to  reduce  high 
risk  program  elements  to  demonstrated 
acceptable  levels  as  well  as  the  cost 
of  doing  so. 

A  risk  assessment  includes  not  only  likelihood 
of  success,  but  also  must  include  the  conse¬ 
quences  of  failure  in  measured  terms,  usually 
dollars,  hence  the  concept  of  a  "cost-risk 
analysis"  becomes  of  interest. 

A  recent  article  by  this  author  (1)  described 
a  statistical  procedure  for  treating  risk 
assessments.  Additional  work  by  the  author  for 
the  Naval  Air  Systems  Command  (2),  and  ongoing 
work  for  the  Air  Force’s  Rome  Air  Development 
Center  further  extend  the  methodology  reported 
In  (3-9).  We  have  developed  a  simple,  yet 
comprehensive,  system  for  not  only  performing  a 
risk  analysis,  but  integrating  the  cost  conse¬ 
quences  into  the  model. 
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The  following  is  a  summary  of  the  steps  in  a 
typical  cost-risk  analysis: 

Step  1;  Identify  Key  Risks 
The  key  risks  or  uncertainties  relevant  to  a 
particular  component/system  and  technology  are 
identified  as  a  first  step  in  assessing  risk.  A 
special  form  was  developed  for  this  step  called 
a  Cost-Risk  Assessment  Input  Data  Sheet. 

Step  2;  Develop  Measurement  Standards 
The  second  step  in  a  risk  assessment  is  the 
definition  process  of  measures  of  effectiveness 
(MOEs),  often  called  measurement  standards.  The 
standards  were  defined  as  far  as  possible  in 
quantitative  terms  rather  than  qualitative  terms. 
Cost  is  only  one  of  several  potential  measure¬ 
ment  standards. 

Step  3:  Develop  an  Analytic  Methodology 
The  methodology  used  in  cost-risk  assessment  for 
combining  several  types  of  risk  is  adapted  from 
the  cost  uncertainty  procedures  described  in 
Refs.  1  and  3. 

The  advantages  of  such  a  procedure  are: 

•  Consistency  across  technologies/ 
manufacturers/program  phases 

•  Ease  of  application  and  explanation 

•  Ability  to  quantify  subjective 
knowledge  of  technical  experts  in 
each  key  risk  area. 

The  essence  of  a  cost-risk  methodology  is 
to : 

•  Estimate  the  distribution  of  cost 
and  risk  associated  with  each  key 
risk  variable  identified  in  Step  1, 
via  three  basic  inputs:  a  low 
value,  a  most  likely  value,  and  a 
high  value. 

•  Apply  a  set  of  simple  analytical 
formulas  to  calculate  the  likeli- 
ood  (probability)  that  any  parties 
value  (such  as  the  goal  or  most 
likely  value)  will  be  exceeded. 

The  objective  is  to  include  in  the  analysis  an 
uncertainty  measure  of  the  costs  required  to 
successfully  accomplish  stated  objectives,  A 
larger  magnitude  of  cost  (mean  or  measure  of 
central  tendency)  implies  that  multiple  design 
alternatives  are  being  retained  which  should 
reduce  development  risk.  A  larger  uncertainty 
(variance  or  measure  of  dispersion)  in  cost  im¬ 
plies  that  potentially  greater  risks  are  being 
taken. 

Little  is  usually  said  about  the  problem  of 
"composite  risk  score"  but  we  shoud  point  out 
that : 


•  Different  risks  (e.g.  technologies) 
may  have  to  be  combined  sequentially 
(in  series)  if  one  development  depends 
on  other  prior  developments. 

•  Competing  technologies  may  be  con¬ 
tinued  in  parallel  to  increase  the 
composite  likelihood  of  success. 

•  One  particular  technology  may  be  more 
crucial  than  others  in  the  series 
(weakest-link-in-the-chain  approach) . 

A  solution  to  the  composite  risk  problem  is  to 
develop  a  "composite  risk  score"  from  subjective 
inputs  of  experts  in  each  technical  area.  For 
example,  our  definition  of  a  "Good"  Risk  in  a 
recent  application  was  "The  technical  approach 
includes  mostly  proven  concepts;  some  minor 
modifications  to  available  equipment  or  technol¬ 
ogy;  risk  is  reasonable  based  on  the  methodology 
employed."  Verbal  definitions  over  a  scale  from 
0  to  100  in  increments  were  used  (Good  was  an  80 
on  this  scale) . 


COST-RISK  ANALYSIS 


The  purpose  of  a  Cost-Risk  System  is  to  provide 
the  acquisition  manager  with  the  quantitative 
cost-risk  values  associated  with  alternative 
acquisition  strategies  at  key  program  milestones. 
This  concept  assumes  that  there  are  fixed  pre¬ 
liminary  design  alternatives  being  developed  on 
a  fixed  time  schdule.  Thus,  the  cost-risk 
assessment  may  be  proposed  either  by  one,  or  by 
multiple  contractors.  These  quantitative  cost- 
risk  values  provide  an  additional  tool  to  the 
acquisition  manager  for  evaluating  the  many 
alternative  strategies  available. 


INPUT  DATA 


For  the  purpose  of  this  cost-risk  assessment  we 
will  use  cost  as  the  measurement  standard  and 
Identify  key  risks  by  system/ subsystem  level  and 
life  cycle  phase. 

In  examining  several  alternatives,  each  contrac¬ 
tor  would  identify  hardware  elements  for  which 
cost-risk  estimates  would  be  prepared.  The 
objective  is  to  isolate  high  design  risk  items 
in  order  to  determine  the  best  strategy  for  ensur¬ 
ing  program  success  recognizing  the  existence  of 
these  elements.  Thus,  the  breakdown  of  each 
weapon  system  should  go  no  lower  than  is  neces¬ 
sary  to  include  hardware  elements  which  are  con¬ 
sidered  to  have  particularly  high  design  risk. 

For  example,  each  contractor  could  consider  the 
following  elements: 

•  Air  Vehicle 

Structure 
Flight  Controls 
-  Other 
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•  Critical  Mission  Systems  (e.g.,  new 

radar) 

•  Propulsion  and  Power 

•  All  Other  (needed  to  add  up  the  total 

costs  by  phase) 

All  cost  elements  should  be  assigned  to  identi¬ 
fied  hardware  elements  of  each  weapon  system. 

The  hardware  elements  for  which  cost  estimates 
are  made  should  be  identical  to  the  elements  for 
which  design  evaluations  are  made.  This  is 
essential  in  order  to  allow  trade-offs  which 
consider  back-up  hardware  elements. 

The  cost  elements  can  be  segregated  by  the  early 
life  cycle  phases  (Concept  Formulation,  Valida¬ 
tion,  and  Full  Scale  Development),  in  order  to 
consider  alternative  programs  in  each  phase  on 
a  comparable  cost  basis. 

Constant  Year  (e.g.  FY80)  costs  should  be  used 
for  consistency.  Note:  the  effects  of  infla¬ 
tion  can  be  separately  handled.  Further,  treats 
ment  of  present  value  is  left  to  the  reader. 

As  an  example  of  the  input  data  collection 
procedure,  assume  an  analysis  will  be  made  of 
one  proposed  configuration  of  V/STOL  "Bj* 

Further  assume: 

•  The  high  technical  risk  elements  are 
the  digital  flight  control  system  and 
the  thrust  deflection  portion  of  the 
propulsion  system, 

•  The  configuration  uses  existing 
(proven  production  configurations  are 
available)  core  avionics  systems, 

•  With  the  above  assumptions  Figure  I 
would  be  prepared.  For  all  hardware 
elements  which  were  not  available  in 
a  production  configuration,  a  code 
will  be  entered  in  each  combination  of 
life  cycle  phase  and  hardware  element. 
The  hardware  costs  must  add  up  to  the 
total  cost.  Thus,  a  column  labeled 
"other1'  may  be  needed. 

•  For  each  code  entered  in  Figure  I  a 
Cost-Risk  Assessment  Input  Data  Sheet 
will  be  completed.  Figure  II  shows  an 
example  of  a  completed  form  for  the 
hypothetical  example.  A  form  like 
this  would  also  be  required  for  the 
"other"  costs. 


ANALYSIS  PROCESS 


A  procedure  is  briefly  described  which  can  be 
used  to  assess  cost-risk  trade-offs  between  both 
alternative  system  concepts  and  program  alterr 
natives  in  acquiring  a  given  concept,  The 
methodology  described  in  this  section  will 
enable  the  decision  maker  to  analyze  the 
following: 


•  The  cost-risk  uncertainties  inherent 

in  alternative  design  concepts  proposed 
by  individual  contractors,  and 

•  The  cost  uncertainties  of  having 
alternative  combinations  of  contractors 
develop  their  respective  design  con¬ 
cepts  in  parallel  (i.e.,  effect  of 
competition) . 

It  is  imperative  to  consider  both  of  the  cost 
characteristics  (magnitude  and  uncertainty), 
hence  a  cost-risk  system  is  developed.  The 
effects  of  competition  can  be  specifically 
treated  in  this  analysis. 

The  procedure  outlined  in  this  section  requires 
several  necessary  steps  before  an  assessment  of 
cost-risk  trade-offs  is  possible.  These  are 
listed  below: 

For  each  contractor's  design: 

•  Develop  a  Probability  Density  Function 
(PDF)  of  cost  (i.e.,  cost  uncertainty) 
for  each  element  of  each  weapon  system 
design  concept  proposed  by  that  con¬ 
tractor.  We  do  this  using  only  the 
Low,  Most  Likely,  and  High  estimates 
of  hardware  element  cost  submitted. 

•  Develop  a  PDF  of  cost  for  each  weapon 
system  design  concept,  i.e.,  for  each 
contractor . 

•  Display  the  cost  uncertainties  graphi¬ 
cally  by  contractor. 

For  each  competitive  chase: 

•  Develop  the  composite  cost  PDFs 
resulting  from  the  parallel  develop¬ 
ment  of  alternative  concepts  (i.e., 
examine  cost-risk  of  1  contractor,  2 
contractors,  etc.,  continuing  the 
development) . 

•  Display  the  composite  cost-risk 
uncertainties  graphically. 

•  Relate  the  resultant  cost-risk  PDFs  to 
each  other  so  that  cost-risk  trade¬ 
offs  can  be  examined. 

Using  these  ground  rules,  the  methodology  accom¬ 
plishes  the  following: 

•  It  treats  the  cost  of  each  hardware 
element/subsystem  as  a  random  variable 
from  a  generalized  Beta  probability 
distribution. 

•  It  develops,  for  each  hardware  element, 
a  cost  Probability  Density  Function 
(PDF)  based  on  a  Low,  Most  Likely,  and 
High  estimate  of  its  cost. 

•  It  statistically  combines  the  hardware 
element  cost  PDFs  (via  an  analytical 
technique  known  as  the  "method  of 
moments")  to  obtain  major  subsystem 
cost  PDFs. 
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•  It  statistically  combines  each  major 

subsystem  cost  PDF  (via  the  same  tech¬ 
nique)  to  obtain  a  total  system  cost 
PDF  for  a  perticular  contractor’s 
design. 

The  cost  uncertainty  of  developing  different 
combinations  of  design  concepts  in  parallel, 
i.e.,  the  effect  of  competition,  can  be  deter¬ 
mined  by  applying  the  "method  of  moments"  tech¬ 
nique  again  to  statistically  combine  total 
system  cost  PDFs. 


EXTENDED  MCR  METHODOLOGY 
(INDEPENDENT  ASSESSMENT  OF  RISK) 


The  procedure  described  above  collapses  cost  and 
risk  into  the  single  measurement  standard  of 
"cost."  However,  in  examining  the  effects  of 
competition,  it  would  be  useful  to  include  an 
evaluation  of  "probable  technical  success." 

The  concept  of  adding  redundancy  to  achieve 
greater  reliability  is  certainly  understood  by 
the  reader.  This  concept  can  be  extended  to  the 
problem  of  assessing  the  effect  of  competition 
in  developing  a  complex  weapon  system,  such  as 
V/STOL. 

Assume  that  it  is  possible  to  derive  some 
measure  of  "anticipated  success"  (i.e.,  the 
probability  of  developing  a  "successful"  weapon 
system)  for  each  design  concept.  In  this  case, 
as  before,  it  could  be  shown  that  by  developing 
two  design  concepts  in  parallel  (i.e.,  adding 
competition),  the  probability  of  achieving  at 
least  one  "successful"  design  is  greater  than 
if  only  one  concept  were  developed. 

It  can  be  seen  then  that  the  effect  of  adding 
redundancy  into  a  system's  design  is  the  same  as 
the  effect  of  adding  competition  to  the  develop¬ 
ment  of  a  weapon  systems  design  concept.  In 
both  cases,  added  redundancy  (or  competition) 
enhances  the  chances  for  success.  However,  in 
both  cases,  this  enhanced  likelihood  of  success 
is  achieved  at  the  expense  of  greater  cost, 


THE  PROBLEM 


The  intent  here  is  to  define  alternative  method¬ 
ologies  that  can  be  employed  by  the  decision 
maker  to  aid  in  making  trader-offs  between  two 
criteria: 

•  The  larger  cost  of  having  alternative 
combinations  of  contractors  develop 
their  designs  throughout  the  various 
stages  of  program  development,  and 

•  Vhe  technical  risk  (or  probability  of 
achieving  a  "successful"  design)  which 
can  be  expected  in  having  each  combi^ 
nation  of  contractors  develop  that 
design. 


THE  SOLUTION 


The  methodology  for  deriving  cost  uncertainty 
for  each  design  concept  can  be  used.  This 
methodology  would  enable  each  contractor  to 
develop  an  uncertainty  distribution  around  cost, 
(i.e.,  a  probability  density  function  (PDF)  of 
cost)  for  their  particular  design  concept.  Using 
the  cost  PDFs  for  individual  contractors,  it  was 
shown  that  composite  cost  PDFs  for  combinations 
of  contractors  could  be  derived  via  the  "method 
of  moments"  technique  (2) . 

Since  individual  contractor  cost  PDFs  do  in  fact 
include  elements  of  design  risk  (i.e.,  larger 
cost  variance  implies  greater  risk),  it  is  pos¬ 
sible  to  perform  cost-risk  trade-offs  for  indi¬ 
vidual  contractors  on  the  basis  of  their  esti¬ 
mated  cost  PDFs  alone.  Unfortunately,  it  is  not 
possible  to  compare  combinations  of  contractors 
solely  on  the  basis  of  cost  uncertainty.  The 
reason  for  this  is  that  cost  uncertainty  is  in 
direct  proportion  to  the  number  of  contractors, 
while  risk  is  inversely  proportional  to  this 
factor.  Figure  III  illustrates  this  point. 

In  Figure  III  cost  risk  profiles  are  illustrated 
using  "most  likely"  system  cost  estimates.  To 
do  this,  it  is  assumed  that  some  measure  of 
anticipated  success  (probability  of  technical 
"success,"  or  TPg) can  be  derived  for  each  con¬ 
tractor’s  design  concept.  Figure  Ill-a  presents 
the  trivial  case  in  which  it  is  assumed  that 
each  contractor  (and  hence,  each  contractor  com¬ 
bination)  has  a  100%  chance  of  developing  a 
"successful"  technical  design  (i.e.,  no  attempt 
to  assess  probability  of  technical  success  is 
made).  In  this  case,  the  decision  maker  would 
never  consider  contractor  combinations,  because 
nothing  would  be  gained  for  the  higher  most 
likely  cost  this  would  require. 

On  the  other  hand,  Figure  Ill-b  illustrates  the 
case  in  which  point  estimates  for  probability 
of  technical  success  are  somehow  derived  for 
each  contractor's  design.  With  this  additional 
information,  the  decision  maker  can  use  the  cost 
PDFs  and  the  cost-risk  profiles  in  trading  off 
the  higher  expected  costs  of  developing  two  in¬ 
dependent  designs  simultaneously  against  greater 
likelihood  of  a  technically  successful  design. 

If  A  has  only  a  60%  probability  of  being  suc¬ 
cessful  and  B  has  only  75%  probability  of  being 
successful,  the  combination  of  A  and  B  should 
give  us  a  90%  chance  of  technical  success  for 
this  particular  acquisition  phase. 

It  is  clear  that,  in  order  to  conduct  effective 
cost-risk  trade-offs  for  combinations  of  con¬ 
tractors,  it  is  necessary  to  estimate  some 
measure  of  the  technical  risk  involved  for  each 
contractor’s  design  concept  in  addition  to  cost 
uncertainty.  Given  that  it  is  possible  to  as¬ 
sess  such  a  figure,  several  alternatives  for 
presenting  cost-risk  trade-offs  to  the  decision 
maker  are  possible. 
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One  simple  alternative  for  providing  cost-risk 
trade-offs  was  discussed  briefly  in  the  previous 
section.  The  methodology  required  two  pieces  of 
information  from  each  contractor: 

•  A  cost  PDF  for  a  particular 
acquisition  phase,  and 

•  An  estimate  of  the  probability  of 
technical  success  (TPs)for  the  design 
concept  and  program  plan  for  that 
phase. 

Given  this  information,  cost-risk  profiles  sim¬ 
ilar  to  the  one  illustrated  in  Figure  Ill-b 
could  be  prepared. 

The  cost-risk  profile  of  Figure  Ill-b  does 
provide  useful  information  to  the  decision  maker, 
however,  it  does  have  one  serious  shortcoming. 
That  is,  for  each  contractor  combination,  there 
is  only  one  point  on  the  cost-risk  profile. 
Presumably,  for  a  given  design  concept,  each 
contractor  will  have  a  set  of  alternative  design 
strategies.  These  design  strategies  will  differ 
for  a  variety  of  reasons,  among  which  are  the 
following : 

•  Some  design  strategies  will  rely  on 
established,  low-risk  technologies 
end  therefore,  will  have  more  certain 
cost  estimates. 

•  Some  design  strategies  will  involve 
more  redundancy  in  the  system  design, 
consequently  driving  system  cost  up. 

•  Some  design  strategies  will  depend 
upon  technological  breakthroughs  and 
hence,  will  have  more  uncertain  cost 
estimates . 

In  brief,  each  design  strategy  of  a  given 
contractor  will  have  a  unique  cost  uncertainty 
and  technical  probability  of  success  (TP  ),  com¬ 
bination  associated  with  it.  Obviously , SFigure 
III  does  not  capture  this  cost-risk  variation 
among  alternative  design  strategies  for  individ¬ 
ual  or  combinations  of  contractors.  Consequent¬ 
ly,  a  methodology  to  deal  with  this  problem  is 
required. 

One  is  tempted  at  this  point  to  use  the  cost 
PDF  developed  for  each  contractor  in  order  "to 
exhibit  the  cost-risk  variation  among  alterr 
native  design  strategies  for  individual  Cor 
combinations  of)  contractors.  However,  we 
caution  against  this!  Consider ,  for  instance, 
Figure  IV,  This  figure  illustrates  the  cost 
PDF  (Pc>  and  its  associated  cumulative  distri¬ 
bution  function  (TP  )  for  one  design  strategy  of 
contractor  A.  On  tSe  cost  curve,  note  that  the 
90th  percentile  cost  (C  )  is  marked.  The  Inter¬ 
pretation  of  this  point  is  the  following:  "The 
probability  that  the  total  cost  for  this  particr 
ular  design  strategy  will  be  less  than  or  equal 
to  CA  Is  0.90."  In  other  words,  0.90  represents 
a  measure  of  cost  uncertainty;  It  does  not  rep¬ 
resent  a  "success"  probability  for  the  technical 
design  strategy.  To  use  It  as  such  is  misleading! 


The  curve  would  be  generated  from  data  on  the 
Cost-Risk  Assessment  Input  Data  Sheets  for  a 
particular  contractor's  design.  The  individual 
estimates  of  Low,  Most  Likely,  and  High  costs 
for  each  cost  element/hardware  element  combina¬ 
tion  are  the  absolute  low,  absolute  high,  and 
most  likely  cost  estimates  for  developing  a 
defined  end  product.  Let*s  assume  the  particular 
end  product  has  a  probability  of  technical  suc¬ 
cess  (TPg)  pf  0.95,  i.e.,  we  think  there  is  a 
954  chance  it  will  work.  Then  the  cumulative 
distribution  function  of  cost  we  developed  above 
should  be  labeled  "For  a  95%  chance  of  technical 
success"  as  shown  In  Figure  V-a . 

The  procedure  described  for  constructing  this 
figure  may  be  repeated  for  several  design  alter¬ 
natives,  i.e.,  other  assumed  values  of  probabil¬ 
ity  of  technical  success.  For  example,  the 
procedure  could  be  repeated  for  a  50-50  chance 
design  (TP  =  0.5)  and  a  high  risk  design  (TP  = 
0.1).  Cost  uncertainty  curves  for  each  of  t^ese 
designs  can  then  be  developed.  Firgures  V-b  and 
V-c  can  then  be  constructed. 

The  data  from  alternative  design  programs, 

Figure  V,  allow  us  to  compare  the  probability  of 
achieving  technical  success  (TP  )  with  various 
probabilities  of  not  exceeding  I  given  cost 
value  (TP  ) .  For  example,  let's  assume  it  Is 
desired  to  determine  the  relationship  between 
probability  of  technical  success  for  various 
values  of  cost  (using  a  cost  value  having  a  90% 
chance  of  being  achieved,  i.e.,  TP  =  0.90).  By 
drawing  a  horizontal  line  at  TP  =  6.90  the  "cost” 
at  three  values  of  technical  success  may  be  ob¬ 
tained  for  contractor  A,  contractor  B  and  the 
combinations  of  contractors  A  and  B.  This  data 
can  then  be  plotted  as  Figure  V-d.  It  is  impor¬ 
tant  to  note  that  although  Figure  V-d  Is  similar 
to  Figures  V-a,  -b,  and  -c,  it  Is  not  measuring 
the  same  physical  phenomenon.  There  is  uncer¬ 
tainty  associated  with  both  cost  and  technical 
success  and  therefore  neither  can  be  determined 
Independent  of  the  other. 

The  data  from  the  curves  of  Figure  V  are  tabled 
below.  One  might  be  tempted  to  use  the  costs 
for  Cases  I,  II,  and  III  as  the  Low,  Most  Likely, 
and  High  costs  of  a  cost  uncertainty  distribu¬ 
tion  directly.  Hopefully  the  analysis  above  has 
pointed  out  the  fallacy  of  doing  this  since  Fig¬ 
ure  V-d  Is  constructed  from  Figures  V-a,  -b ,  and 
-c . 
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Given  three  alternative  design  concepts,  one 
could  construct  an  Acquisition  Strategy  composed 
of,  say  one  Low  Technical  Risk  design;  and  one 
High  Technical  Risk  design.  In  fact,  with  two 
firms  and  three  design  alternatives,  all  nine 
combinations  of  (A, B)  could  be  considered: 

(1.1) ,  (I, II),  (I, III),  (II, I),  (II, II),  (II, III), 

(111. 1) ,  (III, II),  (111,111).  Each  of  these 
would  have  a  composite  technical  probability  of 
success  given  by: 

TP  =  1  - ( 1-TP  )  (  1-TP  ^ ’ 

SAB  sA  sB 

where 

TPS  is  the  technical  probability  of  success 
for  a  particular  case. 

Rigorous  comparison  of  multiple  alternatives 
rapidly  increases  the  complexity  of  this  method- 
ology.  The  use  of  high  speed  computers  enables 
the  acquisition  manager  to  deal  with  this  com¬ 
plexity. 

The  composite  cost  uncertainty  distribution  for 
Che  particular  acquisition  phase  could  be  ob^- 
tained  using  the  MCR  analytical  methodology 
reference. 


CONCLUSIONS 


The  area  of  risk  analysis  is  a  very  complex  but 
increasingly  important  subject.  This  paper  has 
briefly  examined  a  simple  procedure  for  inte¬ 
grating  both  cost  and  technical  risk.  The 
initial  cost-risk  methodology  MGR  developed  is 
being  used  by  several  hardware  contractors  in 
various  risk  assessments  as  well  as  by  ourselves. 

In  the  area  of  cost  estimating  and  risk  assess¬ 
ment  we  believe  uncertainty  should  be  treated 
explicitly.  Several  complex  issues  for  analysis 
still  remain  (such  as  dependence  between  subr- 
sys terns) . 
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Figure  I.  V/STOL  "B"  (CONFIGURATION  #1)  COST  MATRIX  INDEX  CODES 
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AIAA  Acquisition  Strategy  Study 


COST  KATRIX  COOE  - CF°T  COMPILED  3Y  MaxailHan  G.  Gartwingle 

DATE  3/11 /SO _ 

AIRCRAFT  WEAPON  SYSTEM  V/STOL  "3"  Configuration  fl _ 

SYSTEK/SU3SYSTEM  Thrust  Deflection  System _ 

LIFE  CYCLE  PHASE  Validation _ 

COST  UNCERTAINTY  ESTIMATE  ($1,000  Constant  FY1380) 


1.  Low  Value:  $620 

Rationale:  If  Phase  I  full  scale  tests  confirm  the  internal  nozzle 
confi guration  successfully  demonstra ted  by  the  scale 
model  air  flow  tests  completed  during  the  concept  formu¬ 
lation  phase,  the  full  scale  flow  tests  ($60,000)  may 

)he  pr0p0sed  pr°sram.  ($680,000  minus 


Most  Likely  Value:  $680 

Rationale:  A  similar  configuration  successfully  completed  in  1978 
similar  validation  tests  to  those  proposed.  (See  NQ- 
THRUST  Engine  Co.  Report  xxx  of  March  1979.)  The  cost 
of  that  program  ($515,000)  was  escalated  three  years  at 
9.7%  per  year.  ($515,000  x  (1.097)3  =  $680,000  ) 


3.  High  Value:  $820 

Rationale:  Durability  tests  assume  the  final  validation  can  be 
demonstrated  with  only  one  major  redesign.  In  the 

Tt J°f  redesigns  are  required,  an  additional 
will  be  required.  ($580,000  +  $140,000  = 
$320,000.) 


Figure  II.  COST-RISK  ASSESSMENT  INPUT  DATA 
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Figure  IV.  COST  UNCERTAINTY  FUNCTIONS  FOR  DESIGN  A 
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UNCERTAINTY  DIAGNOSIS  AND  EARLY  LEARNING 


Richard  Alan  Goodman 
Graduate  School  of  Management 

University  of  California,  Los  Angeles 

Early  learning  is  a  critical  managerial  func¬ 
tion  which  will  significantly  reduce  both 
schedule  and  costs  overruns.  Appropriate  un¬ 
certainty  diagnosis,  coupled  with  selected 
managerial  action,  will  increase  the  potential 
for  creating  early  learning  in  development 
projects.  In  addition,  early  learning  will 
increase  the  smoothness  of  the  project  manage¬ 
ment  while  at  the  same  time  assuring  appropri¬ 
ate  performance  levels  of  the  desired  product. 
Understanding  this  concept  is  one  of  the  Keys 
to  project  success.  The  early  learning  defi¬ 
nition  and  the  description  of  three  types  of 
uncertainty,  which  follow,  provide  guidelines 
for  managerial  action  (Freeman,  1972;  Twiss, 
1974) . 

KEYS  TO  PROJECT  SUCCESS 

The  study  of  large  scale  development 
projects  has  led  to  a  range  of  phenomena  that 
are  usually  present  in  all  successful  projects. 
These  phenomena  form  the  basis  of  the  following 
discussion.  The  discussion  of  the  individual 
guidelines  will  explore  both  commercial  and 
government  development  projects  and  necessarily 
will  be  general  in  nature. 

MARKET  ORIENTATION.  It  is  clear  from  the 
studies  that  successful  projects  have  a  very 
explicit  customer  orientation.  While  not  de¬ 
creasing  the  need  for  technical  expertise,  the 
successful  projects  reflect  deep  and  detailed 
understanding  of  the  customer's  technical  re¬ 
quirements  and  the  technical  context  of  these 
requirements.  The  personnel  on  such  projects 
work  hard  to  apply  and  adapt  their  expertise 
to  these  needs.  In  general,  the  customer  is 
kept  informed  of  the  progress  and  the  technolo¬ 
gical  development.  This  leads  to  valuable 
exchanges  of  information  which  in  turn  allows 
for  appropriate  project  choices  as  it  becomes 
evident  tluit,  on  some  dimensions,  an  ideal 
solution  may  not  be  possible.  In  other  words, 
these  projects  utilize  a  problem  solving  ap¬ 
proach  rather  than  forcing  a  standardized 
technology  to  fit  a  particular  situation. 

(All  too  often,  technologists  become  so  in¬ 
volved  in  their  approach  that  they  require 
customer  adaptation  rather  than  technological 
adaptation.  A  marketing  orientation  is  quite 
the  opposite.) 

RELEVANCE  TO  CORPORATE  OBJECTIVES.  Strong 
corporate  support  is  an  important  factor  in 
successful  projects  and  increases  in  strength 


when  the  project’s  contribution  to  corporate 
objectives  is  perceived.  As  the  perceptions 
of  such  value  decreases  then  the  tendency  to 
neglect  certain  projects  increases.  Such 
neglect  results  in  reduction  of  a  project's 
priority  and  this  in  turn  results  in  slacken¬ 
ing  of  performance  demands.  The  natural 
outcome  of  such  a  situation  are  missed  mile¬ 
stones  and  poorer  design  quality.  The 
cumulative  effect  of  such  a  tendency  to 
neglect  a  project  is  customer  dissatisfaction 
and  a  final  product  which  is  not  as  close  to 
the  desired  outcome  as  might  have  been  expec¬ 
ted. 

EFFECTIVE  PROJECT  SELECTION  AND  EVALUA¬ 
TION  SYSTEM.  The  likelihood  of  a  project 
being  well  suited  to  the  corporate  objectives 
is  enhanced  by  a  well  designed  project  selec¬ 
tion  and  evaluation  system.  Such  a  system, 
by  its  very  existence,  indicates  that  corpor¬ 
ate  support  is  an  issue.  In  addition,  the 
technical  competence  of  the  system  increases 
the  likelihood  that  projects  selected  will 
develop  in  line  with  the  overall  corporate 
objectives.  Even  a  rather  crudely  designed 
system  will  force  the  project  proposers  to 
explicitly  consider  the  corporate  objectives. 

EFFECTIVE  PROJECT  MANAGEMENT  AND  CONTROL. 
Without  a  good  management  and  control  system 
the  direction  of  the  project  cannot  be  deter¬ 
mined,  corrective  action  cannot  be  taken,  and 
the  outcome  will  be  less  likely  to  match 
original  expectations.  In  addition,  the 
adaptation  that  is  required  to  correct  such 
a  drift  in  the  outcome  cannot  be  anticipated 
and  the  customer  is  often  unable  to  use  the 
product  immediately. 

SOURCE  OF  CREATIVE  IDEAS.  It  may  seem 
rather  obvious  to  state  that  sources  of 
creative  ideas  are  necessary  in  the  develop¬ 
ment  process.  By  their  very  'nature,  engineers 
and  scientists  represent  such  sources.  There 
appears  to  be  significant  distinction  between 
the  organizations  which  rely  upon  their  ex¬ 
isting  technical  resources  and  those  which 
actively  pursue  a  policy  of  enriching  these 
resources.  This  enrichment  may  come  from  a 
recruitment  policy  that  continues  to  bring  in 
fresh  personnel.  Attendance  at  professional 
meetings,  for  example,  will  bring  in  fresh 
ideas.  Some  attention  to  and/or  training  in 
creativity  techniques  is  important.  All  of 
these  approaches  seem  to  be  significant 
features  of  those  organizations  which  are 
able  to  pursue  successful  projects. 

ORGANIZATION  RECEPTIVE  TO  INNOVATION.  A 
management  style  or  climate  which  is  receptive 
to  creativity  is  extremely  important.  Scien¬ 
tists  and  engineers  can  apply  their  talents  in 
a  creative  or  an  analytic  manner.  In  the 
latter  mode,  proposed  solutions  are  analyzed 
for  their  respective  strengths  and  weaknesses 
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and  variations  suggested  to  improve  the  solu¬ 
tions.  This  mode  relies  heavily  upon  the 
quality  of  the  initial  proposed  solution. 

Many  projects  represent  a  simple  or  relatively 
simple  extension  of  an  existing  solution  or 
approach.  In  contrast  to  this,  the  "new  boy" 
approach,  suggested  by  Goodman  and  Abernathy 
(1978),  describes  projects  whose  success  is 
twice  the  industry  average  and  whose  approach 
was  not  a  simple  extension  of  existing  tech- 
nology  but  rather,  a  new  "creative"  concep¬ 
tualization.  When  these  two  issues  are 
addressed  by  management,  a  higher  tendency 
toward  finding  and  employing  creative  ideas, 
as  appropriate,  exists.  This  tendency  seems 
important  in  successful  projects. 

COMMITMENT  BY  GNE  OR  FEW  INDIVIDUALS. 

One  last  factor  which  seems  to  be  associated 
with  project  success  is  the  existence  of  a 
so-called  "project  champion."  It  appears 
that  strong  commitment  by  one  or  a  small 
group  of  individuals  provides  a  drive  and  a 
real  concern  for  the  quality  of  the  outcome. 
This  motivational  force  increases  the  proba¬ 
bility  of  project  success.  This  initiative 
and  concern  usually  ensures  a  more  sophist ica- 
ted  understanding  of  the  issues  (both  techni¬ 
cal  and  customer  requirements)  and  maintains, 
on  the  part  of  the  staff,  a  spirit  derived 
from  their  sense  of  the  project’s  importance. 

Research  on  successful  projects  indicates 
that  in  each  case  these  conditions  all  exist¬ 
ed,  albeit,  in  varying  intensity.  That  is, 
the  successful  project  had  a  market  orienta¬ 
tion  while  at  the  same  time  being  relevant  to 
corporate  objectives.  The  organization 
pursuing  the  project  had  an  effective  project 
selection  and  evaluation  system  and  an  effec¬ 
tive  project  management  and  control  system. 
Creativity  was  enhanced  by  a  commitment  to 
developing  sources  of  creative  ideas  and  a 
management  receptive  to  innovation.  Finally, 
the  project  was  managed  by  an  individual  or 
a  small  team  which  was  very  highly  committed 
to  its  success. 

EARLY  LEARNING 

The  generic  nature  of  research  and 
development  can  be  described  in  several  ways. 
One  very  powerful  concept  is  a  learning  model, 
ihe  research  and  development  process  can  be 
defined  as  the  sequential  and/or  concurrent 
purchase  of  information.  An  idea  is  posed, 
then  an  analysis  is  undertaken  to  understand 
how  the  idea  might  work.  Tests  are  run  to 
gather  data  about  a  component  of  the  idea, 
etc.  Each  project  phase  provides  new  data 
until  the  final  drawing  set  and  the  final  test 
data  are  available. 

This  learning  model  is  represented  by  the 
sketch  in  Figure  1,  where  time  and  information 
are  the  important  issues  in  a  project.  For 


simplicity,  a  project's  objective  is  to  gather 
100%  of  the  necessary  information  for  success¬ 
ful  completion. 


information 

100% 


The  notion  of  a  learning  path  can  then  be 
added  to  the  model.  That  is,  learning  accom¬ 
plished  during  a  project  could  be  plotted  on 
this  graph  and  the  result  is  shown  by  the 
project's  path  from  the  beginning  to  the 
successful  completion  at  100%.  One  of  an 
infinite  number  of  such  paths  is  illustrated 
in  Figure  2. 


Information 

100% 


Figure  2. 

A  LEARNING  PATH 

For  the  purposes  of  this  discussion  two 
opposite  types  of  learning  paths  are  used. 

The  path  on  the  left  of  Figure  3,  represents 
the  late  learning  case.  In  the  former  case, 
the  major  uncertainty  of  the  project  is  re¬ 
duced  in  the  earlier  portions  of  the  project 
and  only  small  problems  need  to  be  overcome  as 
the  project  nears  completion.  In  the  latter 
case,  major  uncertainty  is  encountered  very 
late  in  the  project  which  create  great  diffi¬ 
culty  for  managers  trying  to  bring  the  project 
to  an  orderly  conclusion. 
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100  % 


Figure  3. 

EARLY  AND  LATE  LEARNING  PATHS 


The  basic  thesis  of  this  middle  section 
is  the  proposition  that,  in  any  development 
process,  THE  KEY  MANAGEMENT  TASK  IS  TO  CREATE 
AN  EARLY  LEARNING  PATH.  Guidelines  for  mana¬ 
ger  iul  actions  which  lead  to  early  learning 
arc  contained  in  the  third  major  section  of 
this  chapter.  Before  considering  those 
guidelines,  the  resource  implications  of  the 
early  and  late  learning  paths  will  be  dis¬ 
cussed  . 

The  typical  resource  expenditure  in 
development  work  follows  the  underlying 
project  stages  or  phases.  The  initial  phase 
normally  requires  a  small  number  of  personnel 
to  work  together  very  closely.  As  concep¬ 
tualization  becomes  firmer,  the  project  work 
can  begin  to  include  other  organizational 
members.  The  increasing  rate  of  personnel 
utilization  means  that  the  expenditure  rate 
continues  to  increase  as  research  blends  into 
development  and  development  blends  into  de¬ 
sign,  etc.  Simply  speaking,  more  and  more 
people  can  be  effectively  employed  on  the 
project  as  the  project  develops.  If  the 
phases  were  well  bounded  and  explicit,  this 
increasing  rate  of  expenditure  would  look 
like  a  step  function  similar  to  Figure  4. 
Otherwise  the  increase  would  be  a  bit  more 
smooth. 


Expenditure 

rate 

high 


TABLE  k. 

EX? ENTITLES  RATE  OVER  TIME 


In  general,  the  cost  associated  with  an 
error  will  be  higher  in  the  later  stages  of  a 
project.  This  occurs  because  the  project  de¬ 
velops  more  and  more  collateral  documentation 
as  time  progresses.  An  early  change  in  a 
basic  concept  may  require  only  a  rewrite  to  a 
few  persons.  The  same  change  in  a  basic  con¬ 
cept  later  in  the  project  has  a  more  substan¬ 
tial  effect.  Then,  drawing  sets  and  parts 
lists,  test  and  operations  manuals,  schedules, 
and  many  other  documents  are  affected.  In 
addition,  the  coordination  of  such  a  change 
among  design,  test,  financial,  manufacturing, 
logistics  and  reliability,  etc.  requires  an 
enormous  effort.  Thus,  ceteris  paribus,  a 
change  which  occurs  early  will  be  less  costly 
than  that  same  change  occurring  later  in  the 
project . 

The  value  of  the  learning  path  concept 
can  be  enriched  even  further.  The  learning 
path  position  is  a  measure  of  the  areas  of 
remaining  uncertainty.  As  these  uncertainties 
are  reduced,  changes  are  often  required.  In 
some  cases  the  uncertainty  reduction  simply 
moves  the  project's  original  estimate  of  the 
correct  solution  from  uncertain  to  certain. 

The  more  usual  result  is  the  need  for  a  minor 
or  major  change. 

The  super impo sit  ion  of  the  expenditure 
rate  graph  with  the  learning  path  graph  pro¬ 
vides  an  indication  of  the  likely  cost  impacts. 
The  conclusion  is  rather  obvious.  The  uncer¬ 
tainty  reduction  on  the  early  learning  path 
occurs  at  a  time  when  the  rate  of  expenditure 
is  low  and  thus,  when  the  cost  of  change  is 
low.  In  late  learning  many  of  the  problems 
of  the  project  are  uncovered  during  the  high 
rate  of  expenditure  phases  and  the  concommit- 
ment  change  costs  are  quite  high.  Thus,  the 
reasonable  conclusion  is  that  THE  KEY  MANAGE¬ 
MENT  TASK  IS  TO  CREATE  AN  EARLY  LEARNING  PATH. 

So  far  this  has  been  a  conceptual  discus¬ 
sion.  There  are  many  points  which  can  be 
argued  and  probed  and  otherwise  contested. 

These  points,  however,  would  not  detract  from 
the  general  conclusion  that  early  learning  is 
very  valuable  and  that  it  is  an  important 
management  responsibility  to  try  to  proceed  on 
an  early  learning  path.  To  accomplish  such  a 
result  would  require  omniscience  on  that  part 
of  management.  This  is  clearly  not  realistic. 
The  lack  of  omniscience  does  not  logically 
suggest  that  this  presentation  is  not  useful. 
Instead,  what  is  called  for  are  guidelines 
for  managerial  action  that  allow  the  project 
to  move  toward  the  ideal  early  learning  path. 
That  is,  improvements  in  management  style  are 
possibly  based  upon  both  concept  and  data. 

The  next  section  of  this  chapter  raises  the 
question  of  uncertainty  diagnosis  and  suggests 
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appropriate  managerial  guidelines  to  follow 
the  diagnosis. 

UNCERTAINTY  DIAGNOSIS 

Guidelines  for  managerial  actions  depend 
upon  the  quality  of  the  initial  diagnosis  and 
the  ability  to  carry  out  the  actions  diagnos¬ 
tically  suggested.  To  sharpen  the  manager's 
diagnostic  capability,  three  distinct  cate¬ 
gories  of  uncertainty  are  defined;  TECHNOLO¬ 
GICAL,  USE,  and  PROCESS.  Within  each  category 
the  issues  of  identification,  of  assessment 
and  of  implied  managerial  actions  are  dis¬ 
cussed  . 

TECHNOLOGICAL  UNCERTAINTY.  The  manager 
must  initially  examine  each  of  the  basic 
technologies  which  are  to  be  used  in  the 
development  project.  Of  particular  interest 
is  the  question  of  the  specific  technological 
approach  to  be  used  in  each  of  the  key  areas 
or  subsystems.  The  diagnostic  questions  that 
should  be  asked  about  each  approach  focus  on 
lx>th  the  staff  engaged  in  the  approach  and  the 
technology  itself.  In  the  former  category  is 
the  specific  staff  familiarity  with  the  ap¬ 
proach.  In  the  latter  category  is  the  age  or 
history  of  the  approach,  the  ability  to  use 
analysis  in  problem  solving  and  the  level  of 
intensity  of  previous  work. 

First,  examine  the  selected  approach's 
age.  In  a  number  of  situations,  the  specific 
technological  approach  will  be  a  standard  de¬ 
sign  approach  that  has  often  been  previously 
used.  The  staff  will  certainly  have  to  adapt 
the  approach  to  the  explicit  parameters  of 
the  situation  but  the  approach  itself  is  well 
known.  This  situation  represents  one  end  of 
the  approach  age  spectrum.  At  the  other  end 
of  the  spectrum  is  an  approach  which  has 
been  created  solely  in  response  to  the  current 
presenting  problem.  This  is  an  approach  which 
looks  very  promising  on  paper  but  has  not  been 
tested.  The  managerial  question  here  is 
whether  the  proposed  approach  can  be  reduced 
to  practice  in  the  time  available  for  project 
completion.  The  managerial  alternatives  are 
to  use  an  older  and  more  certain  approach,  if 
it  will  deliver  the  performance  necessary,  or 
to  take  the  risk  and  use  the  new  approach. 
Sometimes,  there  are  middle  ground  approaches 
which  improve  performance  potential  but  still 
reduce  the  uncertainty  of  success. 

A  second  diagnostic  question  focuses  on 
the  ability  of  a  particular  technology  to  be 
analyzed.  Some  technologies  are  clearly  ana¬ 
lytical  and  can  be  rather  well  defined  without 
the  gathering  of  test  data  or  experience. 

Other  technologies  can  only  be  approximated 
by  analysis.  Then  test  must  be  gathered  and 
judgement  applied  to  this  test  data  before  a 
successful  solution  is  found. 


The  third  diagnostic  question  is  an  in¬ 
quiry  about  the  level  of  intensity  of  previous 
work.  A  crude  scale  of  intensity  would  start 
at  the  low  end  with  simple  analysis  and  pro¬ 
ceed  through  component  level  tests  performed 
on  prototypes  or  partial  mock-ups  to  higher 
level  intensities  such  as  tests  of  a  full  pro¬ 
totype  in  the  laboratory.  At  the  highest  end 
would  be  testing  of  the  approach  with  a  pro¬ 
duction  piece  of  equipment  within  the  actual 
use  environment. 

Of  course,  the  existence  of  a  well  de¬ 
veloped  approach  does  not  drastically  reduce 
the  uncertainty,  if  this  approach  is  being 
applied  by  a  new  technical  staff.  Very  few 
complex  ideas  can  be  communicated  for  sophis¬ 
ticated  use  by  simply  reading  about  them. 

The  subtleties  of  any  technological  product 
is,  more  often,  then  not  a  subject  that  can 
only  be  learned  by  experience.  The  so-called 
NIH  (Not  Invented  Here)  phenomena  basically 
arises  when  talented  personnel  attempt  to  use 
an  approach  which  they  have  read  about  hut  not 
experienced.  Thus,  in  addition  Lo  Liu*  basic 
diagnostic  questions  that  relate  to  the  var¬ 
ious  technological  approaches,  one  must  ask 
about  the  specific  experience  of  the  project 
team. 

This  section's  basic  issue  is  the  presen¬ 
tation  of  some  guidelines  for  diagnosis  so 
that  the  development  manager  can  focus  the 
project's  various  technological  uncertainties. 
For  clarity,  the  discussion  has  necessarily 
been  an  approximation  of  the  true  complexity 
of  development.  Enlightened  management  will 
use  these  guidelines  to  identify  their  uncer¬ 
tain  areas  and  to  search  for  ways  to  reduce 
these  uncertainties  in  the  early  phases.  What 
follows  are  some  suggestions  about  appropriate 
actions  that  management  might  take  under  the 
various  conditions  of  uncertainty. 

The  guidelines  here  are  initially  drawn 
from  extensive  examination  of  the  parallel 
strategies  used  in  the  development  of  torpedo 
systems  for  the  United  States  Navy.  (Aber¬ 
nathy,  1979).  This  has  been  extended  into 
commercial  situations  in  the  automobile  indus¬ 
try  (Abernathy,  1978) . 

Under  conditions  of  staff  inexperience, 
it  is  appropriate  to  develop  a  functional 
prototype  early  in  the  project  and  subject  it 
to  simulated  use  tests.  This  provides,  as 
quickly  as  possible,  an  understanding  of  many 
of  the  subtleties  of  the  overall  design  ap¬ 
proach  and  serves  to  more  quickly  reduce  major 
uncertainties.  This  approach  may  not  be  the 
obvious  approach  if  other  conditions  in  the 
technological  area  are  straightforward  but  the 
key  deficiency  that  arises  from  staff  inexper¬ 
ience  can  only  be  helped  by  the  experience  of 
a  rush-to-prototype  test  strategy. 
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When  the  age  of  the  approach  is  ascer¬ 
tained,  several  managerial  actions  may  be 
required.  Early  managerial  intervention  will 
be  the  best  action  if  the  subsystem  approach 
has  too  short  a  history.  As  a  simple  guide¬ 
line,  unless  a  particular  approach  has  been 
under  serious  investigation  for  approximately 
a  year,  then  serious  cause  for  concern  exists 
and  contingency-  plans  should  be  formulated. 

New  approaches  are  usually  selected  when  per¬ 
formance  requirements  are  higher  than  can  be 
promised  by  the  older  approaches.  But,  in 
general,  a  new  approach  has  a  high  probability 
of  not  being  reducible  to  practice  in  time  for 
current  project  needs.  That  is,  the  approach 
may  still  be  sufficiently  uncertain  when  the 
time  comes  to  enter  the  production  phase  that 
serious  delays  and  cost  overruns  are  highly 
1 ikely . 

When  such  a  new  approach  is  selected, 
there  are  high  probabilities  that  the  project 
team  will  eventually  find  it  necessary  to 
fall  back  on  an  older  approach.  This  older 
approach  will  still  have  to  be  adapted  to  the 
current  project.  The  adaptation  cost,  at  a 
late  date,  is  significantly  higher  than  would 
have  occurred  if  the  fallback  approach  had 
been  carried  along  as  a  contingency.  The 
managerial  action  suggested  in  this  situation 
is  to  bring  along  the  older  approach  in  para¬ 
llel  with  the  newer  approach.  As  well  as 
adapting  the  older  subsystem  approach  to  the 
newer  application,  it  is  important  that  the 
larger  system  be  designed  so  that  it  can 
relatively  easily  accept  either  of  the  two 
approaches.  This  enabling  condition  in  the 
design  involves  additional  costs,  but  should 
the  contingency  plan  be  necessary,  these 
costs  would  have  been  incurred  at  the  earlier 
less  expensive  phase  of  the  project. 


results  show  areas  where  the  performance  may 
be  unacceptable  then  the  appropriate  manager¬ 
ial  action  would  be  the  initiation  of  a  test¬ 
ing  program  aimed  directly  at  the  areas  of 
potential  underperformance.  This  two  stage 
approach  would  seem  to  be  the  most  effective 
in  early  reduction  of  the  approach’s  uncer¬ 
tainty. 


Even  when  management  determines  that  the 
technical  approach  has  been  pursued  for  an 
extended  period  of  time,  they  do  not  have  suf¬ 
ficient  cause  for  assuming  a  high  probability 
of  success.  They  still  must  ascertain  the 
approach’s  intensity  level.  The  higher  the 
intensity  level,  the  less  the  uncertainty. 
While  the  parallel  strategy  works  well  for  low 
experience  situations,  it  does  not  help  in  low 
intensity  of  experience  situations.  Here  the 
more  appropriate  strategy  is  to  quickly  devel¬ 
op  a  prototype  and  engage  in  a  testing  program. 
An  increase  of  the  development’s  intensity 
level,  even  with  only  prototype  hardware,  will 
more  quickly  reduce  the  inherent  uncertainty 
of  the  situation  and  still  occur  in  the  low 
cost  early  project  phase. 

To  summarize,  different  situations  sug¬ 
gest  different  strategies.  This  is  captured 
in  Table  1.  In  the  area  of  technological  un¬ 
certainty  certain  diagnostic  questions  can  be 
asked  and  the  answers  can  be  used  to  suggest 
explicit  managerial  action  that  would  serve  to 
reduce  the  uncertainty  of  the  project  in  the 
early  phase  and  create  an  EARLY  LEARNING  PATH. 


STAFF  INEXPERIENCE 
Ru sh-to-pro to ty pe 

LOW  AGE  OF  APPROACH 
Parallel  Strategy 


STAFF  EXPERIENCE 
Normal  process 

HIGH  AGE  OF  APPROACH 
Single  Strategy 


Of  course,  another  possibility  is  to 
refuse  to  use  approaches  which  have  so  little 
history  and  to  select  an  approach  with  a 
high  success  probability.  When  high  perfor¬ 
mance  requirements  are  necessary  this  more 
conservative  strategy  would  seem  to  be  un¬ 
attractive.  But,  then  the  contingency  plans 
suggested  above  become  much  more  important. 


NOT  AN ALY ZABLE  AN ALY ZABLE 

Normal  Testing  Anal/Sens.  Test  Analysis 

LOW  EXPER.  INTENSITY  HIGH  EXPER.  INTENSITY 
Rush-to-Prototype  Analyze 

Table  1. 

STRATEGIES  FOR  TECHNOLOGICAL  UNCERTAINTY 


When  a  computer  or  other  analytic  assis¬ 
tance  can  determine  the  eventual  system  per¬ 
formance  then  the  most  appropriate  uncertainty 
reducing  method  is  the  employment  of  an  ana¬ 
lytic  strategy  in  the  early  phase  of  the 
project.  On  the  other  hand,  when  analysis 
can  only  approximate  the  performance  results 
then  other  alternatives  become  better  for 
early  uncertainty  reduction.  Initially  a 
system  analysis  is  performed,  followed  by  a 
sensitivity  analysis  of  the  results.  The 
sensitivity  analysis  would  indicate  whether 
the  range  of  likely  results  are  acceptable. 

If  they  are  acceptable  then  there  is  no 
particular  problem.  If  the  range  of  likely 


USE  UNCERTAINTY.  When  envisioning  a  new 
product,  process  or  service,  an  idealized 
image  of  the  new  idea's  use  is  created.  This 
simplification  of  the  actual  use  situation  pro¬ 
vides  a  powerful  communication  device  which 
can  ease  the  explanation  of  the  new  concept's 
essential  features.  When  faced  with  developing 
this  concept,  much  attention  is  required  to 
adapt  it  to  the  multifaceted  details  of  actual 
use.  The  less  a  project  team  knows  about  the 
rich  details  of  use,  and  possible  misuse,  the 
more  use  uncertainty  exists.  Here  the  term 
use  uncertainty  is  meant  to  convey  the  level 
of  knowledge  of  the  actual  use  environment. 
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This  uncertainty  is  a  specific  character¬ 
istic  of  the  team  undertaking  the  development. 

The  importance  of  the  use  environment  can 
be  illustrated  with  a  rather  simple  example. 
When  the  procurement  process  is  studied,  cer¬ 
tain  logical  decision  rules  are  evident.  If 
two  proposals  for  a  deep  space  experiment  are 
received  and  they  are  identical  in  technical 
approach  then  there  is  no  basis  for  selecting 
one  over  the  other.  If  the  organizations 
proposing  the  projects  differ  by  the  amount 
of  experience  they  have  in  the  deep  space  en¬ 
vironment,  then  the  one  with  greater  knowledge 
would  clearly  be  selected,  ceteris  paribus. 
This  decision  rule  speaks  to  the  generally 
held  notion  that  a  more  thorough  understanding 
of  the  use  environment  is  a  significant  meas¬ 
ure  of  the  higher  likelihood  of  success. 

Thus,  a  conventional  wisdom  is  clearly  on  the 
side  of  the  importance  of  level  of  knowledge 
of  the  use  environment  or  its  converse,  the 
concept  of  use  uncertainty. 

In  theoretical  terms,  the  work  of  Richard 
Normann  (1969)  provides  another  explanation  of 
this  concept Ts  importance.  He  defines  two 
concepts:  domain  and  distant  environment.  An 
organization’s  domain  is  the  environmental 
sector  with  which  it  has  frequent  interaction. 
Because  of  this  frequent  interaction,  the  or¬ 
ganization  has  developed  a  structure  for  re¬ 
receiving  environmental  information  and  making 
sense  from  it.  This  ability  to  read  subtle 
cues  in  the  environment  allows  the  organiza¬ 
tion  to  proceed  with  relative  sureness  when  it 
is  operating  within  its  domain.  The  distant 
environment  represents  the  remaining  environ¬ 
mental  sectors.  The  key  phenomena  about 
stimuli  arising  in  the  distant  environment  is 
the  organization’s  inability  to  make  sense  of 
the  stimuli.  The  organization  then  must  em¬ 
ploy  a  mediator  (consultant)  to  make  sense  of 
the  stimuli  and  to  inform  the  organization. 

When  the  management  determines  that  high 
use  uncertainty  exists,  then  one  of  two  gener¬ 
ic  approaches  can  be  taken  to  accomplish  early 
uncertainty  reduction.  One  is  a  rush-to- 
prototype  approach  which  quickly  provides  the 
project  team  with  equipment  for  field  test. 

The  field  test  provides  first  hand  knowledge 
of  many  subtle  details  and  possible  problems 
in  the  proposed  specific  design.  The  second 
is  the  recruitment  of  personnel  who  have  de¬ 
tailed  knowledge  of  the  use  environment.  This 
approach  is  similar  to  the  NASA  decision  to 
have  the  astronauts  as  major  members  of  the 
design  teams.  The  astronauts  provided  insight 
and  information  that  was  far  better  than  might 
have  been  provided  by  only  an  analytic  ap¬ 
proach.  When  faced  with  low  use  uncertainty 
management  can  look  to  the  technical  arena  and 
make  appropriate  decisions  based  upon  the  un¬ 
certainties  determined.  High  use  uncertainty 
would  suggest  explicit  rush-to-prototype  or 


recruitment  strategies  even  if  there  were  low 
technical  uncertainty. 

PROCESS  UNCERTAINTY.  This  refers  to  the 
uncertainty  introduced  into  the  development 
processes  by  less  than  ideal  management 
processes.  Given  two  identical  proposals  with 
similar  experience  in  the  use  environment 
there  is  no  logic  for  selection.  But  when  the 
respective  managements  are  considered,  there 
is  often  a  clear  cut  distinction.  Here,  as 
above,  conventional  wisdom  supports  the 
importance  of  process  uncertainty.  It  would 
be  beyond  the  scope  of  this  presentation  to 
detail  the  diagnostic  questions  necessary  for 
ascertaining  the  level  of  process  uncertainty 
in  a  given  situation.  Suffice  it  to  say  that 
planning  and  control  on  one  hand  and  organiza¬ 
tion  on  the  other  would  be  the  natural  focus 
of  such  questions. 

SUMMARY 

In  brief,  this  chapter  has  argued,  that 
THE  KEY  MANAGEMENT  TASK  IS  TO  CREATE  AN  EARLY 
LEARNING  PATH.  It  has  then  followed  this 
argument  with  the  concept  that  uncertainty 
diagnosis  is  indeed  possible  and  that  differ¬ 
ent  findings  would  suggest  different  manager¬ 
ial  actions.  For  this  purpose,  a  set  of 
guidelines  for  diagnosis  and  action  within  the 
three  categories  of  TECHNOLOGICAL  USE  AND 
PROCESS  UNCERTAINTY  were  proposed. 
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ABSTRACT 

Uncertainty  in  DOD  Systems  Acquisition  lies  specifi¬ 
cally  in  the  lack  of  knowledge  characteristic  in 
development  of  Systems  requiring  1  state-of-the-art ' 
technology.  While  perhaps  overly  ambitious  the  only 
structured  way  to  investigate  these  phenomena  is  with 
the  assumption  of  macro  efficiency  and  system 
delivery;  and  the  use  of  a  series  of  macro  models. 
Consideration  of  the  macro  model  leads  to  the  follow¬ 
ing  three  hypotheses:  first,  for  all  firms  as  re¬ 
search  to  sales  increases  cost  of  sales  ratio  de¬ 
creases;  second,  the  above  hypothesis  will  be  more 
pronounced  for  civilian  firms;  third,  civilian  firms 
will  be  more  efficient  than  government  contractors 
in  periods  of  high  inflation  variability. 


INTRODUCTION 


This  paper  lias  evolved  from  a  study  of  the  impact  of 
inflation  and  other  financial  and  managerial  factors 
upon  the  economic  viability  of  firms  in  the  aerospace 
industry. 

The  study  differs  from  other  financial/managerial 
studies  of  the  aerospace  industry  in  that  it  accesses 
a  data  base  that  is  extremely  comprehensive,  namely 
the  publicly-reported  business  and  financial  infor¬ 
mation  covering  a  large  proportion  of  all  prime  as 
well  as  sub-contractor  aerospace  firms.  Further, 
the  methodology  employed  is  unique  in  that  it  seeks 
to  examine  the  financial  strengths  of  firms  by  class 
of  product  or  service  offered,  so  that  defense  capa¬ 
bilities  can  he  assessed  more  thoroughly. 

PART  1  of  the  paper  defines  risk  and  uncertainty  and 
compares  the  two  concepts  in  light  of  tentative  con¬ 
clusions  generated  by  a  review  of  defense  contracting 
over  two  decades. 

PART  II  discusses  preliminary  version  of  an  uncer¬ 
tainty  model  developed  from  the  financial/managerial 
study  of  aerospace  contracting  firms  progresses. 

PART  I 


COMPARISON  OK  RISK  AND  UNCERTAINTY 


1.  Risk 

Risk  may  be  defined  as  a  historically  conceived  phen¬ 
omenon  for  which  a  historical  probability  may  be 
established.  For  instance,  it  is  well-known  fact 
that  all  persons  die  at  one  time  or  another.  Civen 
society's  and  business's  long  experience  with  this 
phenomenon,  certain  tables  have  been  developed  to 
show  that  a  person  at  a  given  age  will,  in  all  proba¬ 
bility,  live  for  n  more  years,  or  will  live  until  Nl. 

Insurance  policies,  written  for  large  numbers  of 
persons  of  a  given  age,  can  "insure"  the  person 
against  a  permature  death — death  prior  to  his  ex¬ 
pected  life  span.  If  the  company  writes  sufficient 
policies,  the  person  who  dies  unexpectedly  at  a 
younger  age  will  find  the  insurance  policy  monies 
will  be  sent  to  his/her  estate,  while  the  company 
recovers/overall  costs  from  all  insurance  promlnums 
plus  investment  earnings. 

The  risk  that  the  person  (any  given  person  or  persons) 
will  die  before  the  given  life  expectency  is  included 
in  the  premium  charged.  There  is  no  doubt  that  the 
person  will  (eventually)  die.  The  only  doubt  is 
WHEN  the  person  will  die.  That  doubt — risk  of  early 
death — could  be  insured  against. 


2.  Uncertainty 

Certainty  or' uncertainty  finds  its  key  in  the  exis¬ 
tence  of  knowledge.  The  relationship  is  inverse  in 
that  as  the  level  of  knowledge  increases  the  degree 
of  uncertainty  decreases.  It  is  precisely  the  fact 
that  uncertainty  is  greatest  when  knowledge  is  at  its 
lowest  level  that  distinguishes  uncertainty  from  risk. 

Uncertainty  and  risk  can  be  seen  clearly  in  the  case 
of  prime  contracting  for  major  weapons  systems, 
designed  to  fulfill  a  specific  mission  and  judged  at 
least  partially  with  reference  to  the  kind  of  mis¬ 
sion.  The  mission  purpose  is  often  found  to  require 
an  advanced  technology  or  high  degree  of  technological 
skill  that  has  not  been  perfected.  The  technology 
required  may  not  be  simply  ' state-of-ar t . '  At  this 
point,  not  knowing  the  design,  production  and  perfor¬ 
mance  problems  required  to  produce  the  system  the 
areas  of  uncertainty  are  highest  precisely  because 
the  level  of  knowledge  regarding  design,  technology 
cost  and  contractor  performance  is  lowest. 
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3.  Department  of  Defense  Views  of  Risk  and  Uncer¬ 
tainty  in  the  Weapons  System  Contracting  Process:  A 
Question  of  Priorities 


Certainty  for  the  Department  of  Defense  in  the  design 
and  acquisition  of  a  weapons  system  would  exist  only 
if  the  Air  Force  decided  to  design  and  let  for  con¬ 
tract  production  and  acquisition  of  a  weapons  system 
already  known,  such  as  the  F4  Phantom.  For  that  kind 
of  weapons  system  there  would  be  certainty  in  the  con¬ 
tracting  process,  and  the  DOD  would  be  in  the  comfort¬ 
able  position  of  being  able  to  manage  the  entire 
process  from  conception  to  operation  and  support. 
Certainty  would  exist  because  there  would  be  complete 
knowledge,  with  the  possible  exception  of  erratic  or 
inflationary  cost  increases. 

Uncertainty,  however,  would  exist  if  the  Air  Force 
were  attempting  to  develop  a  new  system  that  involved 
a  technology  in  advance  of  'state-of-the-art'  tech¬ 
nology.  The  uncertainty  would  exist  because  of  the 
simple  fact  of  lack  of  knowledge.  The  lack  of  know¬ 
ledge  or  uncertainty  would  inevitably  cause  errors. 

In  the  period  before  1962  when  the  Department  of  De¬ 
fense  employed  cost-plus  contractual  arrangements, 
emphasis  was  placed  upon  performance  rather  than  cost 
control.  The  cost  was  increased  or  the  schedule  was 
delayed  until  the  problem  was  solved,  such  that  the 
system  could  perform  at  the  desired  level.  Since 
1972,  with  the  establishment  of  the  incentive  con¬ 
tract,  a  higher  priority  has  been  placed  on  cost 
minimization.  In  the  light  of  present  contracting 
policies  implemented  by  the  Congress,  White  House  and 
Department  of  Defense,  uncertainty  must  be  examined 
in  light  of  cost  minimization. 


4.  Categories  of  Uncertainty  and  The  DOD 

To  this  end,  let  us  examine  this  question  as  it  has 
been  presented  in  the  DOD/FAI  Acquisition  Research 
Symposia.  The  DOD  appears  to  be  concerned  with 
three  types  of  uncertainty  in  the  contracting-acquis- 
tion  process: 

Design  and  technology  uncertainty; 

Uncertainity  regarding  the  ability  of  the 
contractor  to  meet  the  terms  of  the  con¬ 
tract  (scheduling);  and 

Cost  uncertainty. 


a.  Design  Uncertainty 

Uncertainty  with  respect  to  design  often  results  from 
uncertainty  about  technology.  "This  type  of  uncer¬ 
tainty  is  and  probably  will  remain  a  fact  of  life  for 
the  Department  of  Defense.  Defense  contracting  deals 
with  an  unparalleled  rate  of  technological  develop¬ 
ment  which  makes  costly  programs  a  way  of  life."^ 
Technology  problems  have  led  to  60%  of  cost  overruns.-^ 


If  the  rate  of  technological  innovation  and  demand 
by  the  DOD  for  spohistication  continues  as  in  the 
past,  technological/design  uncertainty  will  continue 
to  be  a  major  issue  in  consideration  of  the  con¬ 
tracting  process. 

b.  Contractor  Uncertainty 

The  second  type  of  undertainty  involves  the  ability 
of  the  contractor  to  meet  the  terms  of  the  contract. 
Here,  the  DOD  Is  concerned  with  quality  and  delivery. 
Will  the  contractor  be  able  to  produce  the  product 
as  contracted?  Will  we  get  the  goods  in  time?  Con¬ 
tractor  uncertainty  also  includes  terms  of  mainten¬ 
ance  and,  possibly,  support.  Will  the  contractor  he 
able  to  back  his  product?  Will  the  product  be  useful 
for  an  appropriate  period  of  time? 

c.  Cost  uncertainty 

Cost  uncertainty  arises  because  of  incorrect  projec¬ 
tions  of  increases  in  prices  of  input  factors  such 
as  labor  and  materials.  These  uncertainties  are 
economic  by  nature.  But  cost  uncertainty  may  also 
encompass  the  uncertainties  of  the  previously  men¬ 
tioned  types.  In  other  words,  besides  not -Wing 
certain  what  the  project  will  cost  given  that  we  are 
assured  of  optimal  conditions,  the  other  conditions 
of  uncertainty  also  contribute  to  cost  fluctuation. 

5.  Risk,  Uncertainty  and  The  Goals  of  The  DOD 

Before  outlining  the  DOD’s  approach  to  these  three 
types  of  uncertainty,  it  is  important  that  we  con¬ 
sider  the  goals  of  the  DOD.  The  survey  of  the 
literature  indicates  the  goals  which  are  important: 

1.  Performance-r-having  the  best  or  most 
appropriate  weapon  system.  Here, 
design  and  technology  are  key  factors; 

2.  Scheduling — being  abxe  to  deliver  the 
system  where  and  when  it  is  needed;  and 

3.  Cost  minimization. 

These  goals  are  often  conflicting  and  may  even  be 
exclusive  of  one  another.  For  example,  to  get  the 
best  weapon,  one  may  need  to  wait  for  (X+l)  amount  of 
time,  even  though  the  weapon  is  needed  at  time  (X). 
And  the  best  weapon  at  an  optimal  time  could  be  the 
most  expensive.  It  is  obvious  that  these  conditions 
can  occur  in  the  course  of  the  contract.  How  does 
the  DOD  reach  a  decision?  The  "Goal  Programming 
Approach,"  outlined  in  the  literature,  seems  to  best 
describe  the  DOD  decision  making  process.  The  Goal 
Programming  Approach  seeks  to  satisfy  a  set  of  goals 
(often  conflicting)  which  are  structured  as  con¬ 
straints.  The  importance  of  each  goal  is  indicated 
by  the  priority  placed  on  the  goal.^  The  vital  ques- 


tion  to  be  asked  in  each  specific  case  of  procurement 
is  "What  priority  is  placed  on  each  goal?"  And  "What 
priority  should  be  placed  on  each  goal?"  These  ques¬ 
tions  may  have  different  answers. 

6.  Relationships  Among  Uncertainty,  Goals  and  Cost 
Minimization 

Design/Technology  Uncertainty,  affects  the  goal  of  per¬ 
formance.  To  be  uncertain  of  whether  a  design  if 
appropriate  or  whether  the  technology  will  be  effec¬ 
tive  will  affect  both  the  performance  and  quality  of 
the  system  eventually  delivered.  Thus,  the  reduction 
of  uncertainty  is  design/ technology  would  enhance 
performance.  It  is  therefore  relevant  and  important 
for  DOD  to  deal  with  uncertainty  in  this  respect. 
Similarly,  uncertainty  concerning  the  contractor ' s 
ability  to  meet  contractual  arrangements  affects  the 
goal  of  scheduling.  And  finally,  cost  uncertainty 
affects  the  cost  minimization  goal.  In  addition,  all 
of  the  above  uncertainties  compound  the  effect  on  cost. 
Thus,  the  goal  of  cost  minimization  is  affected  at 
every  stage  of  uncertainty:  "High  systems  cost  and 
cost  growth  (cost  uncertainty)  appear  to  arise  primar¬ 
ily  from  efforts  to  subdue  difficult  technolgy 
(technology  uncer taintv) . . . and  the  acceptance  of 
optimistic  assumptions  about  long  term  predictability 
of  technology  and  the  cost  of  coping  with  it  (all 
three  uncertainties) . ^  Cost  overruns  have  been  docu¬ 
mented  in  the  literature  to  be  as  high  as  243%  over 
original  estimates  and  could  be  even  greater. 

The  design- to-cost  concept  DTC  was  formulated  by  the 
DOD  with  the  specific  objective  of  cost  minimization. 
The  earliest  DTC  literature  studied  (1972)  states  the 
objectives:  "The  DOD  directive  5000.1  has  placed  new 
emphasis  upon  designing  to  a  specific  cost  objective. 
This  concept  of  controlling  costs-to-produce  while  a 
product  is  being  developed  is  supported  with  adequate 
industrial  management  disciplines  which  arc  within 
the  public  body-o f-knowledge . " '  The  procedure  has 
operated  on  the  basis  of  cost  "ceilings."  thus,  it 
seems  that  cost-minimization  has  been  the  primary  goal, 
and  the  performance  goal  had  been  subordinated:  the 
DOD  contracted  for  "The  most  performance  for  the 
approved  cost-function."  But,  as  the  design-to-cost 
concept  has  progressed  through  time  and  experience, 
the  focus  has  shifted.  There  is  still  cost  awareness, 
but  performance  is  rapidly  becoming  the  primary  goal: 

"Today,  government  and  industry  find  them¬ 
selves  operating  in  an  environment  that 
entails  responsibility  for  selecting  design 
concepts,  materials,  and  processes  that,  in 
conbination,  yield  systems  that  require  the 
least  cost  to  acquire  and  maintain  while 
still  achieving  performance  goals. "8 
(See  also  notes  9  -  15) 

7.  Reflections 

Much  of  the  remainder  of  the  discussions  in  the 
Symposia  proceedings  deal  with  how  the  DOD  can  mini¬ 
mize  Liu*  itiu'i'ii  ;i  ini  y  in  the  contracting  process.  As 
long  as  cost  min  Ini  tzal  Ion  is  a  priorlLy  of  the  DOD, 


design  and  performance  uncertainty  will  remain.  One 
way  of  solving  the  ’knowledge  gap*  in  the  areas  of 
technology  is  using  money  for  research.  However, 
even  if  money  were  available  in  unlimited  amounts  for 
contracting,  it  is  not  clear  that  uncertainty  could 
be  eliminated.  When  systems  require  technology  that 
is  in  advance  of  'state-of-the-art'  technology,  uncer¬ 
tainty  will  probably  remain  the  critical  element  in 
DOD  systems  acquisition. 

PART  II 

THE  ASSESSMENT  OF  UNCERTAINTY 


1.  Uncertainty,  Priorities,  and  The  Research  Function 

As  stated  in  Part  I,  uncertainty  forces  decision 
makers  to  resort  to  priorities  in  dealing  with  the 
unknown.  The  point  of  attack  has  been  research.  The 
approach  is  to  employ  pure  research  as  an  aid  in 
setting  priorities  at  the  outset  and  then  to  use  the 
resulting  priorities  in  selecting  the  applied  re¬ 
search  projects  to  help  in  resolving  performance, 
delivery,  and  cost  uncertainties. 

Many  techniques  for  setting  priorities  may  be  involved. 
The  setting  of  priorities  may  encompass  goal  program¬ 
ming,  for  example.  Basic  research  may  employ  the 
DELPHI  technique  as  well  as  scientific  experimentation. 


2.  DOD  Policy  and  The  R  &  D  Expenditure 

Research  expense  (R  &  D)  is  one  of  the  outlays  most 
readily  controllable  in  cost  control  programs,  since 
it  is  a  managed  or  budgeted  cost.  As  a  result,  R  &  D 
expense  has  probably  decreased  as  a  percentage  of 
government  aerospace  contracting  cost,  consequently 
firms  may  have  become  less  efficient  in  meeting 
delivery  and  performance  standards.  Many  problems 
which  could  have  been  detected  early  through  R  &D 
do  not  appear  until  well  into  the  production  or  the 
deployment  stage,  where  they  become  much  more  expen¬ 
sive  to  correct. 

Developers  of  military  aerospace  systems  probably 
confront  more  unknowns  than  do  developers  of  civilian 
aerospace  systems.  For  example,  it  is  probably  more 
difficult  to  discern  the  reprisal  tactics  of  an  enemy 
than  of  a  competitor.  And  the  stakes  are  immensely 
higher,  adding  to  the  pressure  for  correct  decisions. 

Yet  civilian  aerospace  expenditures  for  research  may 
be  higher,  in  proportion  to  sales,  than  is  true  of 
the  military.  There  may  be  just  as  pronounced  a 
relationship  between  the  R  &  D  expenditure  level  and 
the  efficiency  of  the  civilian  aerospace  firm.  One 
significant  factor  in  this  relationship  may  be  capital 
intensity.  Where  proper  equipment  exists,  research 
may  be  more  fruitful  and  may  be  encouraged,  as  could 
be  the  case  more  often  with  non-government  aerospace 
contractors . 
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3.  Efficiency,  R  &  D  and  Uncertainty 

It  is  possible  to  design  a  model  for  investigating 
the  efficiency/R  &  D?  uncertainty  issue  at  an  aggre¬ 
gate  level.  Employing  the  variables  discussed  above, 
the  following  model  may  be  considered: 

COS  *  f  (GSALE,  RESALE,  COR) 

The  variables  are  identified  as  follows: 

COS  Ratio  of  COST  OF  SALES  to  SALES, 

(the  independent  efficiency  variable); 

GSALE  Percent  of  SALES  to  government, 

(independent  profit  variable  distin¬ 
guishing  military  from  civilian 
contractors) ; 

RESALE  Ratio  of  R  &  D  EXPENDITURES  to  SALES 

(independent  variable  denoting  level 
of  research  effort);  and 

COR  Capital  Output  Ratio,  Ratio  of  TOTAL  ASSETS 
to  NET  SALES 

(independent  variable  denoting  capital 
intensity. ) 


The  research  issues  to  be  confronted.  The  central 
issue,  oversimplified  initially,  is  whether  firm 
efficiency  is  related  to  R  &  D  expenditures.  The 
GSALE  variable  allows  us  to  ask  this  question  of 
civilian  contractors  first,  then  of  military  contrac¬ 
tors,  dividing  the  sample  into  the  two  groups,  for 
more  effective  study. 

Our  fundamental  premise  is  that,  if  firms  in  general 
are  efficient  at  the  (macro)  cost  level,  using  COS, 
then  individual  firms  in  that  group  will  also  be 
efficient  in  fulfilling  design  and  contractor  perfor¬ 
mance  requirements  at  the  micro  level.  For  example, 
a  firm  that  meets  customer  demands  for  quality  in 
civilian  markets  or  on  civilian  contracts,  maintaining 
cost  control  and  profitability,  is  also  likely  to 
meet  government  contracting  standards  and  still 
control  costs.  Symbolically,  we  may  express  the 
relationship  as  follows: 

COS  EFFICIENCY  t  More  Effective  Design 
and  Contractor  Perfor¬ 
mance  on  All  (Including 
Government)  Contracts 

Specific  hypotheses  to  be  investigated. 

i.  For  all  aerospace  firms,  military  and 
civilian,  as  RESALE  increases  COS  ratio 
decreases  and  vice  versa.  Symbolically: 

A  COS 

-  <  0 

A  RESALE 


ii.  The  above  relationship  will  be  more  pro¬ 
nounced  for  civilian  contractors  because 
COR  may  be  higher  for  civilian  firms. 
Symbolically : 

A  COS 
— 

jA  RESALE  j  C 

iii.  Civilian  contractors  will  he  even  more 
efficient  than  military  contractors  in 
periods  of  high  variability  in  inflation 
rates,  in  spite  of  the  advantages  of  the 
incentive  type  of  military  contract. 

This  is  because  R  &  D — including  value 
analysis,  will  aid  in  controlling  cost 
increments.  The  civilian  contractor  has 
more  flexibility  in  using  the  R  &  D 
budget  for  cost  control,  (intensifying 
applied  research  and  de-emphasizing  pure 
research  as  conditions  dictate) . 


cos  1 
[a  RES  ALE  _J  M 


The  obvious  overall  issue  is  whether  high — R  &  D — ex¬ 
penditure  firms  have  customarily  delivered  high  qual¬ 
ity,  timely,  low  cost  systems.  We  have  used  a  link¬ 
age  assumption — namely,  that  efficient  firms  at  the 
aggregate  level  usually  deliver  more  effectively. 

This  assumption  bears  further  investigation. 

A  second  complication  relates  to  the  sources  and 
objectives  of  R  &  D  funding.  Two  possibilities  exist. 
First,  firm-sponsored,  general  R  &  D  is  not  necessar¬ 
ily  attributable  to  specific  government  (or  even  to 
specific  aerospace)  projects.  Second,  government- 
sponsored  R  &  D  (say  Milestone  I  research)  is  generally 
attributable  to  specific  government  projects. 

In  order  to  draw  more  realistic  conclusions,  it  may  be 
necessary  to  distinquish  between  sources  and  objec¬ 
tives  of  R  &  D  expenditures  for  each  segment  in  the 
sample . 


PART  III 

SUMMARY  AND  CONCLUSIONS 


The  investigative  approach  described  in  this  paper 
employs  financial  data  relative  to  a  large  sample  of 
both  government  and  non-government  contracting  firms 
in  an  effort  to  assess  uncertainty  in  government  con¬ 
tracting.  Three  elements  of  uncertainty  are  discus¬ 
sed:  contractor  performance,  design/technology ,  and 

cost.  Since  financial  data  can  be  applied  only  to 
the  cost  element,  a  critical  assumption  is  that  firms 
that  control  costs  effectively  are  also  successful  in 
meeting  design  and  performance  criteria.  Since  pro¬ 
ducers  find  themselves  resorting  to  priorities  in 
resolving  uncertainty  issues,  it  is  found  that  re¬ 
search  is  helpful  in  setting  the  critical  priorities. 
Since  those  firms  with  a  more  modern  plant  (including 
laboratories)  may  be  better  equipped  to  do  research, 
capital  intensity  probably  has  a  significant  bearing 
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upon  the  quality  and  usefulness  of  the  research 
carried  out. 

The  factors  discussed  are  combined  into  a  set  of 
hypotheses  that  appear  promising  in  the  assessment  of 
uncertainty.  A  first  hypothesis  states  that  those 
firms  that  control  costs  more  efficiently  engage  in 
research  more  intensively  than  is  true  of  less  effici¬ 
ent  firms.  A  second  hypothesis  is  that  more  highly 
capital  intensive  firms  may  be  better  equipped  to  do 
research  and  that,  consequently,  firms  that  engage  in 
research  more  intensively  are  more  cost  efficient. 

A  Third  hypothesis  states  that,  because  of  the  more 
intensive  research  efforts  of  civilian  contractors, 
they  will  be  more  efficient  than  military  contractors 
in  controlling  costs,  meeting  design  criteria  and 
meeting  performance  criteria  in  periods  of  highly 
variable  inflation  rates.  To  investigate  these 
hypotheses,  the  authors  have  designed  a  research 
model  relating  COST  EFFICIENCY,  (as  the  dependent 
variable),  to  PERCENT  OF  SALES  TO  THE  GOVERNMENT, 
RESEARCH  EXPENDITURE /REVENUE  ratio  and  CAPITAL  INTEN¬ 
SITY  ratio  (as  the  independent  variables).  The 
research  design  contemplates  examining  the  model's 
various  relationships  through  periods  of  both  high 
and  low  inflation  variablity  in  order  to  test  the 
hypotheses  and  to  assess  uncertainty. 
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